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I.  INTRODUCTION 


A.  BACKGROUND 

The  models  and  simulations  (M&S)  of  surface  ship  perfonnance  developed  by  the 
U.S.  Navy  are  expected  to  help  acquisition  program  managers  (PMs),  resource  sponsors, 
and  operational  testers  to  quickly  and  affordably  determine  whether  their  new  or 
modified  warships  are  suitable  and  effective  against  projected  threats.  However,  the  U.S. 
Navy’s  investment  in,  and  use  of,  M&S  for  addressing  critical  operational  issues  (COIs) 
within  a  warship’s  operational  test  (OT)  program  would  not  be  practical  unless  the 
particular  model  or  simulation  was,  on  the  one  hand,  determined  to  be  a  credible 
representation  of  that  which  would  be  physically  tested  and,  on  the  other  hand,  seen  to  be 
risk-acceptable  and  cost-effective. 

The  U.S.  Navy  establishes  the  credibility  of  the  models  or  simulations  it  uses 
through  accreditation  by  a  competent,  recognized  authority.  Commander  Operational 
Test  and  Evaluation  Force  (COMOPTEVFOR)  accredits  the  U.S.  Navy’s  M&S  used  for 
warship  OT  through  its  Verification,  Validation,  and  Accreditation  (VV&A)  process,  as 
documented  in  the  updated  COMOPTEVFOR  Instruction  (COMOPTEVFORINST) 
5000.1  A  of  9  September  2004.  This  instruction  requires  that  all  surface  ship1  PMs,  who 
plan  to  develop  M&S  for  OT,  run  a  standardized  VV&A  process2  that  interacts  with 
COMOPTEVFOR  -  in  both  its  M&S  accreditor  and  operational  tester  functions  -  as  they 
develop  their  M&S. 

In  practice,  however,  few  PMs  have  been  able  to  act  in  full  accordance  with  the 
tenns  of  the  COMOPTEVFOR  process  -  and  yet  their  M&S  are  eventually  accredited, 
though  not  without  a  certain  amount  of  contention  and  political  brinksmanship  among  the 
parties.  Thus,  the  question  arises:  Why  are  U.S.  Navy  surface  ship  acquisition  program 
managers  not  fully  complying  with  the  standardized  VV&A  process  set  forth  by  the 
accreditation  authority?  Both  the  material  developer  and  the  operational  tester  endeavor 

1  This  includes  the  weapons  systems  aboard  surface  ships.  COMOPTEVFOR’s  M&S  VV&A  policy 
applies  as  well  to  the  U.S.  Navy’s  other  warfighting  platforms,  e.g.,  submarines,  aircraft. 

2  COMOPTEVFOR’s  standardized  VV&A  process  will  be  described  and  discussed  in  Chapter  III. 
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to  demonstrate  that  warfighter  requirements  will  be  met  -  in  this  case  through  M&S  -  yet 
each  seems  to  be  looking  differently  at  the  benefits,  costs,  and  risks  entailed  to  achieve 
these  ends.  Thus,  a  practical  VV&A  process  would  need  to  strike  a  realistic  balance 
between  different,  though  legitimately  held,  institutional  attitudes.  The  updated  process 
described  in  COMOPTEVFORINST  5000.1  A  should  represent  progress  in  managing  this 
balance  and  enabling  compliance  to  achieve  the  objectives  of  the  regulation,  cost- 
effective  use  of  models  and  simulations. 

B.  PURPOSE  AND  OBJECTIVES 

This  thesis  endeavors  to  identify  barriers  to  compliance  with  the  VV&A  process 
that  limit  the  ability  to  achieve  cost-effective  use  of  models  and  simulations  for  OT. 

The  objectives  of  this  thesis  are  to:  (1)  identify  the  extent  to  which  surface  ship 
acquisition  program  managers  are  complying  with  the  updated  VV&A  process  prescribed 
by  COMOPTEVFORINST  5000.1  A;  (2)  establish  why  PMs  are  not  fully  complying  with 
the  prescribed  VV&A  process;  (3)  determine  what  factors  help  PMs  fully  comply  with 
the  instruction;  and  (4)  identify  what  improvements  can  be  made  to  the  VV&A  process. 


C.  SCOPE 

This  thesis  will  study  the  VV&A  process  required  by  COMOPTEVFORINST 
5000.1  A  of  9  September  2004  as  it  applies  to  U.S.  Navy  surface  ship  (and  weapons) 
M&S  used  for  OT.  It  will  assess  stakeholder  familiarity  with  the  instruction,  examine  the 
PM’s  current  level  of  compliance,  and  conclude  whether  that  level  is  acceptable.  The 
VV&A  experience  of  recent  U.S.  Navy  surface  ship  (and  weapons)  acquisition  programs 
will  be  sampled.  At  the  heart  of  the  research,  stakeholder  views  will  be  collected  on:  the 
(1)  benefits,  (2)  costs,  and  (3)  risks  seen  in  compliance;  and  the  (4)  benefits,  (5)  costs, 
and  (6)  risks  seen  in  non-compliance.  Finally,  this  thesis  will  consider  potential 
improvements  to  COMOPTEVFOR’s  VV&A  process  that  could  be  embodied  in  a 
revision  to  COMOPTEVFORINST  5000.1  A. 
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The  scope  of  this  thesis  is  intentionally  narrowed  to  address  the  OT  of  U.S.  Navy 
surface  ships  and  their  weapons  systems.  The  research  herein  does  not  cover  M&S  used 
for  non-OT  (e.g.,  developmental  T&E,  resource  requirement  studies,  etc.).  Neither  the 
M&S  used  for  other  U.S.  Navy  warfighting  platforms  (e.g.,  aircraft),  nor  that  used  for 
other  U.S.  military  (e.g.,  U.S.  Anny),  foreign  military,  or  U.S.  civilian  activities  are 
addressed.  Furthermore,  this  thesis  emphasizes  the  accreditation  aspect  of  the  VV&A 
process  over  its  verification  and  validation  steps.  Nevertheless,  the  conclusions  reached 
by  this  thesis,  while  focused  on  U.S.  Navy  surface  ship  (and  weapons)  programs,  are 
intended  for  general  applicability  to  all  weapons  systems,  platforms,  and  capabilities 
acquired  by  the  U.S.  Department  of  Defense  (DoD)  where  M&S  is  used  (and 
consequently  VV&A  is  needed)  to  satisfy  OT  requirements. 

D.  RESEARCH  QUESTIONS 

This  thesis  considers  a  primary  and  four  subsidiary  research  questions. 

1.  The  Primary  Research  Question  Posed  By  This  Thesis  Is: 

What  are  the  barriers  to  compliance  with  COMOPTEVFOR’s  VV&A  process  that 
U.S.  Navy  surface  ship  (and  weapons)  acquisition  programs  face? 

2.  The  Subsidiary  Research  Questions  Are: 

a.  What  benefits  and  liabilities  are  seen  by  the  stakeholders  in 
subscribing  to  the  COMOPTEVFORINST  5000.1  A  VV&A  process,  if  its  standards  were 
consistently  followed  by  all  U.S.  Navy  surface  ship  (and  weapons)  acquisition  programs? 

b.  What  are  the  cost  and  schedule  considerations  that  impact  U.S. 
Navy  surface  ship  (and  weapons)  acquisition  programs  in  the  course  of  complying  with 
the  COMOPTEVFORINST  5000.1  A  VV&A  process? 

c.  What  are  the  risks  observed  by  the  stakeholders  when  the 
COMOPTEVFORINST  5000.1  A  VV&A  process  is  not  followed? 


3 


d.  What  actions  can  be  taken  by  stakeholders  to  ensure  compliance 

with  COMOPTE VF OR's  VV&A  process? 

E.  METHODOLOGY 

The  thesis  research  and  analysis  proceeded  through  seven  steps: 

1.  Stakeholder  Identification 

Major  stakeholders,  who  participate  in  this  arena  of  the  Navy’s  VV&A  process, 

are  identified  and  grouped  by  category.  The  major  stakeholder  groups  that  were 
identified  are:  (a)  Tester/Accreditor;  (b)  Resource  Sponsor;  (c)  Program  Manager;  and  (d) 
VV&A  Standards  Official. 

2.  Literature  Review 

Documents  that  provide  information  and  data  on  the  practices  and  expectations  of 
VV&A  are  identified  and  reviewed. 

3.  VV&A  Forums 

The  researchers  attended  and  participated  in  five  forums  (i.e.,  technical  working 
group  meetings),  sponsored  by  the  U.S.  Navy’s  VV&A  standards  community,  to  obtain 
assistance  in  the  identification  of  active  stakeholders,  current  issues,  and  relevant 
literature.  Forum  attendance  also  provided  opportunities  to  interview  stakeholders. 

4.  Semi-structured  Interviews 

The  researchers  conducted  fourteen  semi- structured  interviews  of  participants 
representing  the  major  stakeholder  groups  for  VV&A  of  M&S  used  for  OT  of  newly 
acquired  (or  modified)  U.S.  Navy  surface  ships  and  their  weapons  systems. 
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5.  Data  Analysis 

Data  from  the  semi-structured  interviews  was  analyzed  and  reported  for  each  of 
ten  interview  questions  by  VV&A  stakeholder  category  and  management  level. 


6.  Conclusion  Development 

Conclusions  were  developed  by  a  systematic  review  of  the  results  against  the 
primary  and  subsidiary  research  questions.  The  conclusion  topics  are  as  follows: 

a.  Extent  of  compliance  with  COMOPTEVFORINST  5000.1  A 

b.  Barriers  to  compliance  with  COMOPTEVFORINST  5000.1  A. 

c.  Factors  for  compliance  with  COMOPTEVFORINST  5000. 1  A. 

d.  Possible  improvements  to  COMOPTEVFORINST  5000.1  A 

7.  Recommendation  Formulation 

Recommendations  were  formed  through  consideration  of  the  conclusions  and  the 
researchers’  general  experience  and  expectations  for  the  future  of  VV&A.  Topics  for 
further  studies  are  identified. 

F.  ORGANIZATION 

Chapter  II  provides  background  information  on  M&S  that  describes  DoD 
strategies  and  how  M&S  is  used  by  the  U.S.  Navy  to  support  surface  ship  acquisition. 

Chapter  III  provides  background  information  on  VV&A  that  describes  DoD 
processes,  COMOPTEVFORINST  5000.1  A,  and  other  VV&A  processes  used  by  U.S. 
Navy  surface  ships  program  offices. 

Chapter  IV  describes  the  research  methodology  used  in  this  study. 

Chapter  V  analyzes  the  results  of  the  data  collected  from  the  interviews. 

Chapter  VI  provides  conclusions,  which  answer  the  thesis  research  questions,  and 
offers  recommendations  and  topics  for  further  studies. 
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II.  BACKGROUND  ON  MODELING  AND  SIMULATION 


A.  INTRODUCTION 

This  chapter  discusses  the  M&S  strategies  of  the  U.S.  Department  of  Defense 
(DoD)  and  the  U.S.  Department  of  the  Navy  (DoN)  and  provides  an  overview  on  the  role 
of  modeling  and  simulation  (M&S)  in  addressing  the  operational  test  (OT)  of  U.S.  Navy 
surface  ships  and  how  M&S  is  being  used  to  resolve  Critical  Operational  Issues  (COIs) 
within  OT. 

B.  DOD  M&S  STRATEGY 

1.  Introduction 

DoD  has  directed  its  Service  Components  (i.e.  the  Army,  Navy,  Marine  Corps, 
and  Air  Force)  to  use  M&S  to  reduce  time,  resources,  and  risks  of  acquisition,  to  increase 
the  quality  of  systems  acquired,  and  to  improve  system  performance.  [1]  In  the  post  Cold 
War  era  of  declining  defense  budgets,  reduced  manning  and  rapidly  evolving  technology 
and  threats,  M&S  can  provide  tailored  test  environments  that  extend  test  envelopes  to 
augment  live  testing.  [1]  These  benefits  are  obtained  only  if  the  M&S  are  credible 
representations  of  the  tested  systems,  projected  threats,  and  the  test  environments.  [1] 

The  widespread  use  of  advanced  M&S  is  necessary  if  the  DoD  is  to  cost- 
effectively  maintain  readiness,  accomplish  its  operational  missions,  make  optimal 
investment  decisions,  and  achieve  dramatic  acquisition  savings.  [2]  To  this  end,  DoD  has 
established  a  strategy  to  foster  simulation  interoperability  and  reuse,  calling  for  them  to 
work  together  as  a  community,  in  systems  of  systems.  [2] 

In  the  late  1980  and  early  1990s,  DoD  and  Congress  began  to  comprehend  the 
vast  potential  of  advanced  M&S  to  revolutionize  the  way  the  Department  made  decisions 
and  conducted  its  operations.  [2]  DoD  has  used  M&S  in  many  diverse  ways  through 
most  of  its  history,  but  advances  in  information  technology  capabilities  and  the  advent  of 
the  distributed  interactive  simulation  concept,  evidenced  by  projects  such  as  Defense 
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Advanced  Project  Agency’s  (DARPA)  Simulation  Network  (SIMNET),  clearly  signaled 
that  major  increases  in  simulation  capabilities  were  possible.  [2] 

Encouraged  by  several  Defense  Science  Board  studies  pointing  to  the  potential 
benefits,  DoD  decided  it  should  make  the  use  of  advanced  simulation  a  corporate  priority. 
[2]  In  the  summer  of  1991,  the  Deputy  Secretary  of  Defense  put  a  single  individual,  the 
Under  Secretary  of  Defense  for  Acquisition  (now  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  (USD  (AT&L)),  in  charge  of  the  Department’s 
simulation  efforts.  [2]  At  the  same  time,  the  USD  (AT&L)  established  a  flag-level 
Executive  Council  on  Modeling  and  Simulation  (EXCIMS)  and  stood  up  the  Defense 
Modeling  and  Simulation  Office  (DMSO)  as  a  new  organization  within  the  Office  of  the 
Director  of  Defense  Research  and  Engineering  (DDR&E).  [2]  DMSO  was  designated  the 
focal  point  for  M&S  matters  across  DoD.  This  management  structure  was  subsequently 
codified  in  DoD  Directive  5000.59,  “DoD  Modeling  and  Simulation  (M&S) 
Management,”  of  January  4,  1994.  [2] 

In  March  1992,  the  EXCIMS,  chaired  by  the  DDR&E,  crafted  a  vision  for  DoD 
M&S.  [2]  This  vision  called  for  the  establishment  of  readily  available,  operationally 
valid  synthetic  environments  to  support  a  full  range  of  application  areas  (e.g.,  joint 
training  and  doctrine  development,  fonnulation  and  assessment  of  operational  plans,  and 
support  of  the  acquisition  process  and  force  structuring).  [2]  The  vision  also  stated  that 
these  environments  would  be  constructed  from  affordable,  reusable  components  that 
interoperate  through  open  systems  architecture.  [2] 

The  vision  called  for  achieving  interoperability  and  M&S  reuse  as  a  fundamental 
strategy  to  satisfy  DoD’s  simulation  needs.  Since  it  was  not  possible  to  build  just  a  few 
simulations  to  satisfy  the  broad  needs  of  DoD,  and  because  any  simulation  would  be  an 
abstraction  of  the  real  world  driven  by  the  goals  of  its  sponsor,  there  would  need  to  be 
many  different  simulations  that  certainly  could  not  be  individually  planned  and  controlled 
by  one  central  office.  [2]  However,  at  the  other  extreme,  the  cost  of  building  narrowly 
focused,  “stove-piped”  simulations  to  satisfy  each  of  the  ever-expanding  simulation 
requirements  would  be  too  high.  [2] 
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The  DoD  M&S  community  appreciated  that  cost,  time,  and  risk  reduction  benefits 
would  come  from  reusing  existing  simulations  to  address  new  requirements  and  flexibly 
connecting  existing  representations  together  to  create  new  simulation  environments.  [2] 
Demands  for  the  reuse  of  simulations  came  soon  and  frequently,  often  even  before  their 
development  was  complete,  but  it  proved  difficult  to  achieve  the  necessary 
interoperability.  [2]  The  key  technical  question  was  how  to  facilitate  interoperability  and 
reuse.  [2]  The  Distributed  Interactive  Simulation  (DIS)  and  Aggregate  Level  Simulation 
Protocol  (ALSP)  interoperability  Standards,  which  had  emerged  in  the  early  1990s, 
represented  major  advances  in  achieving  interoperability,  but  each  applied  to  only  a 
particular  application  domain,  had  significant  technical  limitations,  and  could  not  satisfy 
the  broad  simulation  interoperability  and  reuse  goals  of  the  DoD.  [2]  A  new  approach 
was  required. 

As  these  lessons  were  learned,  DoD,  led  by  the  DDR&E,  established  a  new,  more 
comprehensive  strategy  to  foster  broad  interoperability  and  reuse.  [2]  The  analogy  was 
drawn  to  planning  a  city,  calling  for  simulations  to  work  together  as  a  community,  in  a 
system  of  systems.  [2]  To  foster  a  community  of  simulations  capable  of  being  composed 
into  federations  and  addressing  whatever  requirements  emerge,  requires  a  common 
technical  framework  (CTF)  and  a  set  of  common  services.  [2] 

This  “city  planning”  strategy  was  captured  in  the  first  DoD-wide  M&S  Master 
Plan,  DoD  5000. 59P,  which  was  signed  in  October  1995  after  formal  coordination  across 
all  DoD  Components.  [2]  The  M&S  Master  Plan  (MSMP)  identified  six  objectives 
necessary  to  achieve  DoD’s  M&S  vision.  [2]  The  first  and  most  important  of  the 
objectives  calls  for  the  establishment  of  a  CTF,  to  which  individual  simulation 
developments  must  conform.  [2]  With  this  strategy,  DoD  expects  to  foster 
interoperability,  reusability,  cost-effectiveness,  and  greater  capability.  [2] 

2.  Common  Technical  Framework 

The  CTF  consists  of  three  parts,  the  most  important  of  which  is  the  high-level 
architecture  (HLA).  [2]  An  “architecture”  defines  the  major  functional  components, 
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design  rules,  and  interfaces  for  a  computer-based  simulation  system.  [2]  It  specifies 
(conceptually)  how  they  hook  together  and  work  together  as  a  whole  and  is  distinct  from 
the  software  that  is  required  to  implement  it.  [2]  The  other  two  legs  of  the  CTF  are  a 
common  understanding  of  the  real-world  (tenned  a  conceptual  model  of  the  mission 
space,  or  CMMS)  and  data  Standards.  [2] 


a.  High  Level  Architecture 

The  HLA  has  been  developed,  tested,  and  established  as  the  standard 
technical  architecture  for  all  DoD  simulations.  [2]  DoD  policy  had  required  simulations 
to  comply  with  the  HLA  by  1  October  2000,  be  retired,  or  obtain  a  waiver  from  the  USD 
(AT&L).  [2]  As  of  1  Aug  1997,  approximately  403  simulations  had  been  committed  to 
HLA  compliance.  [2] 

The  HLA  was  developed  by  consortium  of  major  simulation  programs.  [2] 
It  was  chartered  by  EXCIMS,  led  by  DMSO  and  supported  by  govermnent,  academic, 
and  commercial  sponsors.  [2]  In  taking  this  approach,  DoD  emulated  the  commercial 
industry  Standards  development  processes,  which  was  a  significant  innovation  for  DoD 
and  one  of  the  major  ingredients  for  the  success  of  the  HLA  enterprise.  [2]  It  allowed 
broad  community  participation  in  the  HLA  development  process,  fostered  ownership  of 
the  HLA,  and  expanded  the  technical  experience  base.  [2] 

The  HLA  is  not  software,  but  a  set  of  rules  and  specifications  that 
prescribe  how  simulations,  live  systems  (e.g.,  command  and  control  systems,  weapon 
systems  on  instrumented  ranges),  and  supporting  utilities  such  as  data  collectors  and 
wide-area  viewers  work  together  as  a  federation.  [2]  The  HLA  separates  the 
representations,  provided  by  simulations  or  live  systems,  from  the  general 
interconnection  services  that  is  provided  by  a  Runtime  Infrastructure  (RTI).  [2] 

Three  things  define  the  HLA:  rules,  an  interface  specification,  and  an 
object  model  template.  [2]  The  ten  HLA  Rules  establish  the  requirements  for  federation 
components  (term  federates)  and  the  federation  as  a  whole.  [2]  The  HLA  Object  Model 
Template  (OMT)  specifies  a  standard  fonn  in  which  federates  and  federations  must  be 
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described.  [2]  Finally,  the  HLA  Interface  Specification  describes  the  ways  simulations 
interact  with  the  supporting  RTI  software  during  a  federation  operation.  [2] 

The  HLA  is  garnering  much  international  and  commercial  interest.  [2] 
The  North  Atlantic  Treaty  Organization  (NATO)  has  established  a  high-level  steering 
group  on  M&S  that  is  evaluating  the  HLA  as  a  potential  NATO  Standard  to  facilitate 
simulation  interoperability  and  reuse.  [2]  The  Simulation  Interoperability  and  Standards 
Organization  (SISO)  has  begun  the  process  of  establishing  the  HLA  as  an  Institute  of 
Electrical  and  Electronics  Engineers  (IEEE)  Standard.  [2]  The  DARPA  Synthetic 
Theater  of  War  (STOW),  with  United  Kingdom  participation,  is  using  the  HLA  and 
pushing  technology  boundaries  in  a  number  of  important  areas,  including  scalability, 
command  and  control  representation,  and  representations  of  the  natural  environment.  [2] 
Many  foreign  nations  have  begun  HLA-based  simulation  developments.  [2] 

b.  Conceptual  Models  of  the  Mission  Space 

Conceptual  Models  of  the  Mission  Space  (CMMS)  is  a  first  abstraction  of 
the  real  world  activities  associated  with  a  particular  mission.  [2]  Such  conceptual  models 
provide  entities,  actions,  tasks,  and  interactions  (EATI)  representation  of  the  military 
mission  space  that  is  independent  of  any  specific  computer-aided  software  engineering 
(CASE)  tool  or  utility  employed  to  capture  it.  [2]  These  functional  descriptions  are 
intended  to  serve  as  a  representational  resource,  capturing  information  about  various 
military  operations,  to  be  used  by  simulation  developers  and  others.  [2]  This  information 
is  derived  from  authoritative  sources,  is  described  using  common  syntax  and  semantics, 
and  is  independent  of  any  particular  simulation  implementation.  [2] 

Development  and  sharing  of  the  CMMS  is  a  responsibility  shared  among 
various  organizations  in  DoD.  [2]  Warfighters  act  as  the  authoritative  source  for  how  the 
world  works,  specifying  mission-essential  task  lists  and  doctrine.  [2]  The  developers  of 
the  simulation,  cooperatively  and  collectively,  provide  the  knowledge  for  acquisition.  [2] 
DMSO  develops  the  database  management  system:  (1)  providing  knowledge  acquisition 
teams  with  technical  support  (e.g.,  common  semantics  and  syntax,  data  interchange 
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formats);  (2)  registering  the  resulting  CMMS  data  provided  by  the  simulation  developers: 
and  (3)  allowing  wide  access  to  the  CMMS  database  as  it  is  fdled.  [2] 

Other  related  DMSO  efforts  such  as  the  Modeling  and  Simulation 
Resource  Repository  (MSRR)  and  the  Data  Standards  Work,  provide  the  infrastructure 
and  technologies  needed  to  support  CMMS.  [2]  The  MSRR  provides  CMMS  access  to 
simulation  developers,  doctrine  developers,  trainers,  and  other  interested  parties.  [2]  The 
CMMS  database  management  software  supports  a  variety  of  structured  views  for  the 
display  and  manipulation  of  these  conceptual  models.  [2]  A  Data  Interchange  Format 
(DIF)  Standard  has  also  been  developed  for  the  conversion,  integration,  storage,  and 
extraction  of  these  conceptual  models.  [2] 

c.  Data  Standards 

DMSO  led  the  effort  to  develop  M&S  data  Standards  as  directed  by  the 
DoD  MSMP.  [2]  DMSO’s  data  engineering  efforts  have  four  thrusts.  [2]  The  first  is  the 
development  of  Common  Semantics  and  Syntax  (CSS)  and  associated  DIFs.  [2]  CSS  are 
the  logical  structure  and  content  (meaning)  of  any  specific  model  or  data  element.  DIFs 
are  the  physical  representation  (e.g.,  Backus-Naur  Form,  Structured  Query  Language 
(SQL),  bits  and  bytes)  of  data  that  programmers  employ  to  interchange  complex  data 
fields.  [2] 

The  second  thrust  of  the  data  Standards  efforts  is  the  identification  of 
authoritative  data  sources  (ADS).  [2]  ADS  tells  developers  and  users  where  to  go  for  the 
best  data.  Each  military  service  is  identifying  the  appropriate  ADS  associated  with  its 
varied  responsibilities  (e.g.,  weapon  system  characteristics  and  performance,  order  of 
battle)  and  providing  that  information  to  DMSO  for  incorporation  in  the  MSRR.  [2] 

The  third  thrust  is  the  establishment  of  standard  Data  Quality  practices  to 
provide  a  means  to  ensure  databases  are  complete  and  coherent.  [2]  The  fourth  thrust  is 
the  establishment  of  Data  Security  practices  (DS)  to  guide  the  sensitive  matter  of  data 
access  and  release.  [2] 
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Data  Standards  are  intended  to  facilitate  the  creation,  management,  and 
exchange  of  information  data  sources,  simulation  developers,  and  simulation  users  for  all 
phases  of  simulation  operations  -  development,  initialization,  runtime  interchange,  and 
post-execution  review.  [2]  They  are  employed  in  the  drafting  of  HLA  Simulation  Object 
Models  (SOMs)  and  Federation  Object  Models  (FOMs),  CMMS,  and  in  the  authoritative 
representation  of  the  natural  environment,  units,  systems,  and  human  behavior.  [2] 

3.  Common  Services 

A  broad  range  of  common  services  is  being  developed  to  support  both  users  and 
developers  of  M&S.  [2] 

a.  Modeling  and  Simulation  Executive  Agents 
To  provide  focus,  coordination,  centers  of  excellence,  cost  avoidance,  and 
broad  support  to  the  M&S  community  in  common  and  general  use  areas,  the  DoD  has 
designated  four  M&S  Executive  Agents  (MSEAs)  to  serve  the  broad  M&S  community. 
[2]  They  represent  the  three  domain  areas  of  the  natural  environment  (terrain,  oceans, 
aerospace)  and  a  combined  domain  of  foreign  forces  and  joint  national  intelligence 
processes.  [2] 


b.  Natural  Environments 

The  second  major  objective  of  the  DoD  M&S  Master  Plan  (MSMP)  is  to 
provide  timely  and  authoritative  representations  of  the  natural  enviromnent,  which  are 
subdivided  into  the  terrain,  oceans,  and  aerospace  domains.  [2] 

(1)  Terrain.  In  April  1995,  the  USD  (AT&L)  designated  the 
Director,  Defense  Mapping  Agency  (DMA)  as  the  DoD  MSEA  for  Authoritative 
Representation  of  the  Terrain  Natural  Environment.  [2] 

(2)  Oceans.  The  Secretary  of  the  Navy  was  designated  the  MSEA 
for  the  Ocean  Environment  in  April  1996. 
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(3)  Aerospace.  The  Secretary  of  the  Air  Force  was  designated  the 
MSEA  for  the  Air  and  Space  Natural  Environments  in  April  1996.  [2] 

c.  Foreign  Forces  and  U.S.  National  &  Joint  Intelligence  Processes 

The  Director  of  the  Defense  Intelligence  Agency  (DIA)  is  the  MSEA  for 

the  Representation  of  Foreign  Forces  and  U.S.  National  and  Joint  Intelligence.  [2] 

d.  Communication  Services 

Most  simulation  executions  take  place  within  a  single  facility  or  usually 
involve  only  limited  point-to-point  wide  area  communications.  [2]  Additionally,  these 
distributed  simulations  may  require  multi-cast  and  bandwidth  reservation  services  which 
are  not  generally  available  within  either  commercial  or  defense  communication  networks. 
[2]  The  Defense  Infonnation  Services  Network  (DISN),  DARPA  and  the  Defense 
Information  Systems  Agency  (DISA)  have  cooperatively  put  in  place  the  necessary 
communication  capabilities  required  to  support  simulation  activities  that  include 
enhanced  Internet  Protocol  (IP)  services.  [2] 

e.  Verification,  Validation,  and  Accreditation  (VV&A) 

The  DoD’s  strategy  for  VV&A  is  discussed  in  Chapter  III. 

f  Modeling  and  Simulation  Resource  Repository 

The  Modeling  and  Simulation  Resource  Repository  (MSRR)  is  a  set  of 
resources  stored  on  a  distributed  network  of  computers,  and  linked  by  special 
applications  software  and  world-wide  web  (WWW)  protocols.  [2]  The  purpose  is  to 
catalogue  information  and  provide  a  means  for  identifying  and  distributing  reusable 
resources.  [2] 
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g.  Information  Resources 

There  are  two  human-in-the-loop  information  resources,  i.e.,  help  desks, 
available  to  the  DoD  M&S  community.  [2]  The  first  is  the  Defense  Modeling  Simulation 
and  Tactical  Technology  Infonnation  Analysis  Center  (DMSTTIAC),  which  is  a 
government-funded  agency  that  provides  scientific  and  technical  information  and 
analysis  services.  [2]  It  serves  the  M&S,  special  Operations,  test  and  evaluation,  and 
tactical  warfare  communities.  [2] 

The  second  is  the  Modeling  and  Simulation  Operational  Support  Activity 
(MSOSA),  which  is  a  contractor-staffed  activity  operating  under  the  direction  of  DMSO. 
[2]  Its  mission  is  to  assist  DoD  activities  in  meeting  their  M&S  needs  by  providing 
operational  advice  and  facilitating  access  to  M&S  infonnation  and  assets.  [2] 


C.  U.S.  NAVY  M&S  STRATEGY 

1.  Introduction 

The  use  of  M&S  in  the  U.S.  Navy  is  greater  now  than  ever  before.  [3]  Its 
(simulation)  ability  to  represent  a  platfonn  or  process  abstractly  improves  the  U.S. 
Navy’s  capacity  to  train  its  personnel,  plan  and  conduct  operations,  develop  and  procure 
weapon  systems,  and  analyze  force  employment  alternatives.  [3] 

The  major  U.S.  Navy  simulation  programs  are  structured  in  terms  of  their  areas  of 
application,  the  requirements  they  address,  the  capabilities  they  provide,  and  the  funding 
they  receive.  [3]  Major  programs  are  defined  as  those  being  well  recognized  within  each 
community.  [3]  Simulations  are  an  integral  part  of  analysis,  training,  education,  and 
acquisition  processes;  and  as  a  technology,  the  use  of  M&S  is  pervasive.  [3]  In  spite  of 
these  challenges,  providing  a  simulation-specific  baseline  is  critical.  First,  a  baseline 
articulates  simulation  use  and  investment  that  allows  U.S.  Navy- wide  priorities  to  be 
refined.  [3]  Next,  it  describes  application  and  funding  trends,  enabling  U.S.  Navy  leaders 
to  promote  needed  technology  and  resource  expenditures.  [3]  Finally,  it  supports 
simulation  planning  and  documents  key  trends  and  issues  requiring  remedies.  [3] 
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2.  DoN  Modeling  and  Simulation  Vision  and  Goal 

The  U.S.  Navy’s  vision  is  that  M&S  “will  be  a  pervasive  tool  for  operational  units 
and  will  support  analysis,  training,  and  acquisition  throughout  DoN.”  [4]  The  U.S.  Navy 
M&S  vision  calls  for  the  use  of  world-class  models,  simulations,  and  simulators  as  tools 
to  meet  the  future  challenges  of  supporting  force  readiness,  structure,  and  transfonnation. 
[4] 

The  goal  of  the  U.S.  Navy’s  M&S  community  is  to  provide  capable,  reactive,  and 
flexible  simulation  systems  that  are  always  available  to  improve  operational  decision 
making,  assessment  and  experimentation,  training,  and  acquisition.  [5]  Attainment  of 
this  goal,  which  supports  the  vision,  will  improve  the  ability  of  all  U.S.  Navy  decision 
makers  to  execute  their  missions  and  accomplish  their  objectives.  [5] 

3.  DoN  M&S  Organizations 

a.  Navy  Modeling  and  Simulation  Office 

SECNAV  Instruction  5200. 38A,  Department  of  the  Navy  Modeling  and 
Simulation  Management,  established  the  Navy  Modeling  and  Simulation  Office 
(NMSO),  formerly  known  as  the  Navy  Modeling  and  Simulation  Management  Office 
(NAVMSMO),  in  the  Office  of  the  Chief  of  Naval  Operations.  [6]  NMSO  is  responsible 
for  coordination  and  integration  of  M&S  efforts  across  all  U.S.  Navy  warfare  and  support 
areas.  [4]  The  U.S.  Navy  uses  and  relies  on  M&S  in  four  pillars:  assessment,  training, 
acquisition,  and  support  to  operations.  These  areas  cross  organizational  boundaries  and 
represent  the  many  interests  and  responsibilities  of  the  U.S.  Navy.  [4] 

b.  U.S.  Navy  M&S  Standards  Steering  Group  (MS3G) 

The  MS3G  has  representation  from  the  U.S.  Navy  Secretariat  Office,  the 
Chief  of  Naval  Operations,  Fleets,  Systems  Commands,  laboratories,  Federally  Funded 
Research  and  Development  Centers  (FFRDCs),  and  University  Affiliated  Research 
Centers  (UARCs).  [6]  This  broad  participation  and  team  effort  ensures  that  candidate 
Standards  are  rigorously  evaluated  and  appropriately  promoted.  [6]  Participation  in  the 
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MS3G  is  limited  to  Government  personnel  and  their  designated  representatives.  [6]  The 
mission  of  the  MS3G  is  to  oversee  the  operation  of  the  U.S.  Navy  M&S  Standards 
Process,  rather  than  act  as  a  technical  deliberation  body.  [6]  Technical  reviews  of 
candidate  Standards  are  delegated  to  specialized  subgroups.  [6] 

c.  Subgroups 

The  MS3G  is  responsible  for  oversight  of  all  areas  of  the  Navy  M&S 
Standards  Project.  [6]  Technical  expertise  is  provided  by  the  M&S  Standards  Subgroups, 
each  focused  on  a  specific  area  (applications,  such  as  logistics;  or  technologies,  such  as 
data  or  VV&A).  [6]  These  subgroups  are  permanent  in  nature;  each  subgroup  is  chaired 
by  a  member  of  the  MS3G  and  is  appointed  by  the  Director,  Navy  M&S  Standards 
Project.  [6]  Membership  in  M&S  Standards  Subgroups  is  open  to  representatives  from 
government,  industry  and  academia.  [6]  The  M&S  Standards  Subgroup  is  responsible  for 
a  U.S.  Navy  Standard  from  the  time  the  need  is  identified  to  the  time  it  is  approved, 
promulgated  and  advocated.  [6]  These  are  the  subgroups: 

(1)  Application  Planning  and  Review  Groups  (APRGs).  APRGs 
identify  specific  application  needs  for  potential  U.S.  Navy  M&S  Standards,  and  represent 
experts  in  functional  areas  of  the  U.S.  Navy  M&S  Community.  [6] 

(2)  Technology  Area  Groups  (TAGs).  TAGs  address  needs  as 
they  are  identified,  in  the  appropriate  area  of  technology,  by  researching,  analyzing,  and 
reviewing  M&S  Standards  submissions.  [6] 

(3)  Special  Interest  Groups  (SIGs).  When  the  need  for  a  Standard 
is  identified  and  there  is  no  existing  APRG  or  TAG  to  address  this  need,  a  Special 
Interest  Group  (SIG)  will  be  established  by  the  MS3G  Chair.  [6]  The  responsibilities  of  a 
SIG  are  the  same  as  that  of  an  APRG  or  TAG,  as  detennined  by  the  MS3G  Chair.  [6] 
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4. 


U.S.  Navy  M&S  Administrative  Procedures 


a.  MS3G  Meetings 

A  quorum  for  the  MS3G  is  defined  as  a  simple  majority  of  the  voting 
members  of  the  MS3G  (more  than  one  half).  [6]  If  a  member  organization  fails  to  attend 
two  consecutive  meetings,  the  MS3G  Chair  shall  notify  the  organization  that  it  will  lose 
its  right  to  vote  and  be  placed  on  an  inactive  membership  list  unless  represented  at  the 
next  meeting.  [6]  If  the  organization  is  not  then  represented,  it  shall  be  considered 
inactive  and  a  nonvoting  member.  [6] 

b.  Voting 

MS3G  voting  occurs  on  a  nominated  Standard  through  the  use  of  an  online 
web  system  and  only  in  the  5th  Stage  of  the  Navy  M&S  Standard  Process.  [6]  The  voting 
pool  is  dynamic  and  unique,  as  detennined  by  the  MS3G  (or  the  MS3G  chair)  for 
proposed  Standards  undergoing  review.  [6] 

c.  Legal  Issues 

Because  of  copyright  laws,  copyrighted  Standards  submitted  for  review 
cannot  be  posted  publicly.  [6]  Licensed  copies  of  copyrighted  material  used  by  members 
of  the  review  team  are  obtained  separately.  [6]  Proposed  Standards  not  already  approved 
by  a  recognized  Standards  organization  can  still  be  considered  as  U.S.  Navy  M&S 
Standards  candidates  and  U.S.  Navy  M&S  users  can  still  use  them  to  meet  their  needs. 
[6]  If  a  specific  Standard  identified  by  the  U.S.  Navy  M&S  community  is  later  approved 
as  a  government  or  DoD  Standard,  it  will  be  identified  as  approved  by  a  different 
authority.  [6] 


d.  File  Retention 

The  Standards  Coordinator  (SC)  is  responsible  for  maintaining  active  and 
archival  files  for  the  MS3G  meeting  proceedings  and  any  other  document  produced  by  or 
for  the  MS3G  in  the  conduct  of  U.S.  Navy  M&S  Standards  activities.  [6]  These  files 
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shall  be  retained  for  a  minimum  of  five  years  and  shall  include  the  following:  (1)  meeting 
announcements;  (2)  published  agendas;  (3)  meeting  minutes;  and  (4)  meeting 
presentations  (project  and  guest  briefings).  [6] 

5.  U.S.  Navy  M&S  Processes  for  Standards  Review  and  Approval 

The  vision  of  the  U.S.  Navy  M&S  Standards  Project  is  to  promote  a  common  set 
of  Standards  (supporting  protocols,  techniques,  best  practices,  etc.)  for  the  use  and  reuse 
of  U.S.  Navy  models,  simulations,  and  data.  [6]  It  is  not  intended  to  create  a  set  of 
written  Standards  that  remain  unused  in  the  U.S.  Navy  M&S  Standards  repository.  [6]  It 
is  important  to  apply  a  process  that  permits  the  U.S.  Navy  M&S  community  to  keep  pace 
with,  rather  than  react  to,  technological  advances.  [6] 

The  U.S.  Navy  M&S  Standards  conceptual  process  consists  of  three  key 
activities:  (1)  Nominate,  (2)  Evaluate,  and  (3)  Advocate  U.S.  Navy  M&S  Standards.  [6] 
Essential  to  these  key  elements  are:  automated  web  tools;  collaboration  and  facilitated 
support  to  address  and  resolve  issues;  and  M&S  experts  to  review,  leverage,  and  refine 
the  appropriate  Standards.  [6] 

a.  Types  of  Standards 

The  U.S.  Navy  M&S  Standards  process  supports  the  submission  of 
nominated  Standards  that  are  recommended  in  particular  technical  and  application  areas 
as  well  by  the  type  of  Standard  the  submission  provides.  [6]  An  online  submission  form 
is  available  in  the  online  Standards  Nomination,  Evaluation,  Advocacy  and  Central 
Repository  System  (SNEACRS)  for  submitting  a  potential  U.S.  Navy  M&S  Standard.  [6] 
Immediately  upon  submission,  the  nomination  is  assigned  a  system-generated  number  for 
tracking  purposes.  [6]  The  nomination  is  now  called  a  Standards  Needs  Document 
(SND)  and  is  identified  throughout  the  system  by  this  assigned  number.  [6] 

b.  Standards  Review  and  Approval  Process  Stages 

The  execution  of  the  U.S.  Navy  M&S  Standards  process  requires  an 
understanding  of  the  flow  of  activities,  their  relationships,  dependencies,  and  associated 
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data.  [6]  The  U.S.  Navy  M&S  Standards  process  is  broken  down  into  eight  stages  (or 
milestones),  which  are:  [6] 

(1)  Initial  submission 

(2)  Evaluation  by  MS3G  Chair  and  Primary  Subgroup 

(3)  Internal  review  by  NMSO  and  MS3G 

(4)  U.S.  Navy  Community  Review 

(5)  Determine  Disposition  -  MS3G  Vote 

(6)  Determine  Disposition  -  Director,  NMSO  Approval 

(7)  Promulgate  and  advocate  Standard 

(8)  Periodic  review  by  Primary  Subgroup 

6.  U.S.  Navy  M&S  Standards  Project  Charter 

a.  Purpose 

The  purpose  of  the  U.S.  Navy  MS3G  is  to  promote  a  common  set  of 
Standards  for  the  use  and  reuse  of  U.S.  Navy  models,  simulations,  and  data;  as  well  as 
supporting  protocols,  techniques  and  processes.  [6] 

b.  Authority 

The  U.S.  Navy  MS3G  was  established  under  the  authority  of  the  Navy 
Modeling  and  Simulation  Management  Office  (NAVMSMO)  (OPNAV  N6M)  on  8  Nov 
99. [6] 


c.  U.S.  Navy  M&S  Standards  Project 

The  U.S.  Navy  M&S  Standards  Project  is  structured  to  make  every  effort 
to  ensure  that  appropriate  M&S  Standards  are  available  when  needed  for  U.S.  Navy 
M&S  development,  acquisition,  deployment,  and  life-cycle  support.  [6]  It  also  ensures 
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that  the  necessary  infrastructure  is  present  and  applied  to  the  U.S.  Navy  M&S  process. 

[6]  Standards  endorsed  by  this  process  may  be  promulgated  as  mandatory  or  advisory 
(recommended).  [6]  The  Project  identifies  and  establishes  a  strategy  for  achieving  future 
U.S.  Navy  M&S  Standards  evolution.  [6]  To  ensure  that  appropriate  U.S.  Navy  M&S 
Standards  are  available  when  needed  by  the  U.S.  Navy  M&S  community,  frequent  and 
effective  interactions  are  required  among  the  various  participants  in  the  U.S.  Navy 
Standards  Process.  [6] 

d.  Participants 

(1)  Director,  NMSO.  Sponsors  and  provides  requirements  and 
policy  guidance  for  the  project.  [6] 

(2)  Project  Director.  Chairs  the  U.S.  Navy  MS3G  and  is 
appointed  by  the  Director,  NMSO.  [6] 

(3)  The  U.S.  Navy  M&S  User  Community.  Produces  and  utilizes 

M&S  Standards.  [6] 

(4)  The  U.S.  Navy  M&S  MS3G.  Comprised  of  government 
personnel  (and  their  designates),  as  well  as  representatives  from  Federally  Funded 
Research  Centers;  each  NMSO-approved  organization  provides  an  empowered  voting 
member  to  the  Steering  Group.  [6] 

(5)  The  U.S.  Navy  M&S  Standards  Subgroups.  Comprised  of 
M&S  subject  matter  experts  from  government,  industry,  academia,  and  laboratories  as 
well  as  individual  analysts.  This  Subgroup  is  focuses  on  Technology  and  Application 
Planning  and  Review.  [6] 

(6)  Other  U.S.  Navy  M&S  Projects. 

(7)  Platform  and  Weapons  Systems  Acquisition  Programs. 

(8)  OPNAV,  DoN  Secretariat,  Fleets,  and  Systems  Commands. 

(9)  Advisors  and  liaisons  from  the  other  Services’  M&S  Offices 
and  other  DoD  M&S  related  agencies. 
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7.  U.S.  Navy  OT  for  Surface  Ships 

The  SURFACE  Warfare  Division  of  the  Operational  Test  and  Evaluation  Force 
(OPTEVFOR)  is  responsible  for  the  planning  and  execution  of  operational  test  and 
evaluation  of  U.S.  Navy  SURFACE  ships  and  associated  engineering,  auxiliary,  and 
combat  systems.  [7]  This  section  discusses  the  M&S  approaches  of  the  LPD-17,  DDG- 
51,  and  DD-21  (DD-X)  SURFACE  ships  warfare  programs.  The  VV&A  processes  of 
these  three  programs,  as  well  as  the  COMOPTEVFORINST  5000.1  A  process,  are 
discussed  in  Chapter  III. 

a.  LPD-17 

The  SAN  ANTONIO  Class  (LPD-17)  Amphibious  Transport  Docks  is  the 
product  of  many  excellent  ideas  furnished  by  hundreds  of  U.S.  Navy  and  U.S.  Marine 
Corps  fleet  operators,  maintainers,  and  trainers  through  the  design  for  ownership  process. 
[8]  The  12  ships  of  the  LPD-17  Class  will  replace  four  retiring  amphibious  ship  classes. 
[8] 

These  are  versatile  ships  that  perform  the  mission  of  amphibious 
transports,  amphibious  cargo  ships  and  older  dock  landing  ships,  by  incorporating  both  a 
flight  deck  and  a  well  deck  that  can  be  ballasted  and  de-ballasted  to  support  a  landing 
craft.  [9]  The  LPD-17  ships  have  increased  vehicle  and  cargo  carrying  capacity  that 
make  them  a  key  element  of  the  21st  century  Expeditionary  Strike  Groups  (ESG).  [9] 
These  ships  integrate  the  latest  in  shipbuilding  and  warfighting  technologies  to  support 
current  and  future  Marine  Corps  aircraft,  Expeditionary  Fighting  Vehicles  (EFVs),  and 
Landing  craft  air  cushion  (LCAC)  or  conventional  landing  craft.  [9] 

The  LPD-17  program  acquisition  strategy  has  relied  heavily  on  the  use  of 
technology  as  a  means  to  define  requirements,  explore  and  evaluate  design  alternatives, 
test  and  evaluate  the  design,  streamline  production  sequencing,  mitigate  risk,  and 
improving  program  cost  and  schedule  control.  [10]  The  design  review  process  involved 
assessment  of  the  ship  design  using  two  and  three-dimensional  graphic  visualizations, 
generated  from  the  LPD-17  product  model.  [10]  The  visualizations  display  a  design 
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zone’s  structure  and  systems,  and  are  used  to  study  space,  shape,  and  arrangement.  [10] 
The  graphic  visualizations  also  evaluate  the  installation  sequence,  interference,  lines-of- 
sight,  and  accessibility  for  the  purpose  of  design,  assessment,  engineering  analysis, 
planning,  and  instruction.  [10] 

b.  DDG-51 

The  ARLEIGH  BURKE  Class  (DDG-51)  of  multi-mission,  battle  force 
capable  guided  missile  destroyers  provides  precision  engagement  of  targets  ashore,  full¬ 
dimensional  protection  of  joint  and  allied  forces,  and  dominant  maneuver  in  the  open 
oceans  and  littorals  around  the  globe.  [11]  DDG  51s  are  being  constructed  in  flights  to 
incorporate  technological  advancements  during  construction:  Flight  I  (DDG-51  to  71), 
Flight  II  (DDG-72  to  78)  and  Flight  IIA  (DDG-79  to  107)  [12] 

In  May  1998,  the  DDG  51  Program  Office  (PMS  400)  agreed  to  accredit 
all  M&S  used  in  the  Flight  IIA  Live  Fire  Test  &  Evaluation  (LFT&E).  [12]  The  plan 
addressed  LFT&E  critical  issues  that  answered  the  specific  M&S  that  will  be  used,  and 
how  that  M&S  will  be  used  to  address  the  LFT&E  critical  issues.  [12]  M&S  will  be  used 
to  model  the  ship,  identify  characteristics  of  weapons  likely  to  be  encountered  in  combat, 
predict  the  likelihood  of  being  hit,  the  effects  of  hits  and  close  detonations,  and  likely 
locations.  [12] 

LFT&E  requirements  for  the  DDG-51  program  will  be  in  accordance  with 
U.S.  Code,  Title  10,  Subtitle  A,  Part  IV,  chapter  139,  Section  2366;  “Survivability  and 
Lethality  Testing”.  [12]  LFT&E  consists  of  “realistic  survivability  testing”  to  detennine 
the  system  vulnerability  against  threats  likely  to  be  encountered  in  combat  (conventional 
threats  only).  LFT&E  must  be  done  early  enough  in  the  program  to  allow  any  design 
deficiencies  to  be  corrected  before  proceeding  beyond  low-rate  initial  production  (or 
equivalent  program  milestone).  [12] 
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c.  DD-X  (formerly  DD-21) 

In  November  2001,  DoD  announced  that  the  DD-21  program  for  the  future 
surface  combatant  had  been  revised  and  would  now  be  known  as  DD-X.  [13]  The 
program  focus  would  now  be  on  a  family  of  advanced  technology  SURFACE 
combatants,  rather  than  a  single  ship  class.  [13] 

DD-X  replaces  the  DD-2 1  ZUMWALT  program,  which  was  for  a  class  of 
32  multi-mission  destroyers  to  replace  OLIVER  HAZARD  PERRY  class  frigates  (FFG- 
7)  and  SPRUANCE  class  destroyers  (DD-963)  from  2012.  [13]  Unlike  previous  classes 
of  destroyers,  which  were  primarily  to  counter  deep  water  threats,  the  DD-21’s  primary 
mission  would  be  to  provide  land  attack  support  for  ground  forces  and  also  to  carry  out 
traditional  destroyer  missions  of  anti-air,  anti-surface  and  undersea  warfare.  [13] 

The  M&S  vision  for  the  DD-2 1  program  was  to  employ  M&S  throughout 
the  DD-2 1  development  process  to  provide  insight  during  system  engineering  activities, 
support  trade  studies  and  risk  reduction  activities,  and  support  verification  of  “as 
designed”  system  perfonnance.  [14]  M&S  will  also  be  utilized  throughout  the  system 
life  cycle.  [14]  The  Smart  Product  Model  (SPM)  would  be  the  key  to  achieving  this 
vision,  where  the  weapon  system  would  be  conceived,  designed,  built,  tested,  training 
and  operation  would  be  provided  in  a  computer  before  cutting  metal  and  through  the  life 
cycle.  [14] 

The  SPM  is  the  digital  version  of  the  ship  that  is  comprised  of  the  product 
model  data  and  perfonnance  and  behavior  characteristics.  [14]  The  SPM  will  be 
developed  in  three  phases: 

(1)  Phase  I.  Develop  a  Capability  Specification.  [14] 

(2)  Phase  II.  Develop  and  deliver  a  system  prototype.  [14] 

(3)  Phase  III.  Tailor  SPM  for  use  in  manufacturing  and  life  cycle  support. 

[14] 
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D.  CONCLUSION 

This  chapter  discussed  DoD  and  U.S.  Navy  M&S  strategies  and  provided  an 
overview  on  the  administration  of  M&S  in  the  U.S.  Navy.  It  also  addressed  how  M&S  is 
being  used  for  three  surface  ship  programs.  The  next  chapter  provides  background  on  the 
VV&A  processes  prescribed  by  DoD  and  the  U.S.  Navy  and  discusses  the  VV&A 
processes  of  COMOPTEYFORINST  5000.1  A  and  three  surface  ship  program  offices. 
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III.  BACKGROUND  ON  VERIFICATION,  VALIDATION,  AND 

ACCREDITATION 


A.  INTRODUCTION 

This  chapter  provides  background  on  the  Verification,  Validation,  and 
Accreditation  (VV&A)  processes  prescribed  by  DoD  and  the  U.S  Navy.  A  review  is 
made  of  DoD’s  principal  VV&A  instruction  and  of  their  recommended  practices  guide. 
Regarding  the  U.S.  Navy,  the  VV&A  processes  of  COMOPTEVFOR  and  three  surface 
ship  program  offices  are  described. 


B.  DOD  VV&A  PROCESSES 

1.  Introduction 

In  1994,  as  a  result  of  the  great  need  for  credible,  cost-effective,  and  efficient 
M&S,  the  DoD  directed  each  of  the  Services  to  develop  a  VV&A  process.  [1]  As 
recounted  in  the  Hall,  Conwell  &  Stutzman  paper,  “Focusing  on  Credibility  and 
Confidence  -  U.S.  Navy  Modeling  and  Simulation  Verification,  Validation,  and 
Accreditation,”  the  DoD’s  mandated  VV&A  process  was  intended  to  provide  the  end 
user  with  information  that  would  determine  whether  and  how  the  results  of  a  model  or 
simulation  can  be  applied  to  decisions  that  would  ultimately  affect  the  warfighter.  [1] 

In  April  1996,  DoD  Instruction  (DoDI)  5000.61  on  “DoD  Modeling  and 
Simulation  (M&S)  Verification,  Validation,  and  Accreditation  (VV&A)”  was 
established.  This  instruction  implemented  policy,  assigned  responsibilities  and 
prescribed  procedures  for  the  VV&A  of  DoD  M&S.  [1]  In  November  1996,  the  Defense 
Modeling  and  Simulation  Office  (DMSO)  followed  up  with  the  publication  of  the  DoD 
Verification,  Validation,  and  Accreditation  Recommended  Practices  Guide  (RPG)  to 
provide  instructions  on  how,  when  and  under  what  circumstances  formal  VV&A 
procedures  should  be  used.  [1] 
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2.  DoD  Applicability  and  Policy 

As  put  forth  in  the  instruction,  DoDI  5000.61  is  applicable  to  members  of  the 
Office  of  the  Secretary  of  Defense  (OSD),  the  Military  Departments,  the  Chainnan  of  the 
Joint  Chiefs  of  Staff  (CJCS),  the  Combatant  Commands,  DoD’s  Office  of  the  Inspector 
General,  the  Defense  Agencies,  the  DoD  Field  Activities,  and  any  other  organizational 
entities  in  DoD  (that  are  considered  to  be  “DoD  components”).  [15]  All  M&S  that  is  used 
to  support  DoD  activities  are  under  this  instruction,  including  those  that  support 
Operational  Test  and  Evaluation  (OT&E),  which  are  subject  to  guidance  from  the  OT&E 
director.  [15] 

DoDI  5000.61  prescribes  policy  that  states: 

Models  and  simulations  used  to  support  major  DoD  decision¬ 
making  organizations  and  processes  . . .  shall  be  accredited  for  that 
specific  purpose  by  the  DoD  Component  M&S  Application 
Sponsor.  [15] 

The  instruction  also  prescribes  that: 

Each  DoD  Component  shall  be  the  final  authority  for  validating 
representations  of  its  own  forces  and  capabilities  . . .  and  shall  be 
responsive  to  the  other  DoD  Components  to  ensure  its  forces  and 
capabilities  are  appropriately  represented.  [15] 

3.  DoD  Responsibilities 

DoDI  5000.61  outlines  the  responsibilities  for  four  key  positions  within  DoD, 
which  include: 

a.  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics,  who  is  required  to  coordinate  with  the  DoD  components,  develop  policies, 
plans,  procedures,  and  DoD  issuances  for  effective  implementation  and  management  of 
VV&A  of  DoD  M&S;  [15] 

b.  the  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications,  and  Intelligence,  who  is  required  to  resolve  validation  issues  and 
ensure  intelligence  processes  and  capabilities  are  represented  appropriately;  [15] 
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c.  the  Principal  Staff  Assistants  (PSAs)  and  the  Heads  of  the  DoD 
components,  who  are  required  to  plan  and  provide  resources,  as  needed,  to  carry  out  the 
functional  VV&A  responsibilities  in  accordance  with  the  component  priorities;  [15]  and 

d.  the  Chairman  of  the  Joint  Chiefs  of  Staff,  who  establishes  the 
VV&A  policies,  procedures,  and  guidelines  to  satisfy  the  needs  of  joint  activities  that 
reports  to  him/her,  and  who  coordinates  with  other  DoD  components  to  establish 
procedures  for  the  validation  and  accreditation  of  joint  M&S  used  for  joint  applications. 
[15] 


4.  DoD  Procedures 

DoDI  further  states  that  DoD  procedures  require  that  V&V  shall  be  incorporated 
into  the  development  and  life-cycle  management  process  for  all  M&S  that  are  currently 
being  used  by  DoD.  [15]  The  procedures  also  require  that,  in  performing  V&V  for  any 
federation  of  models,  the  elements  must  be  physically  connected  to  exchange  data  and 
should  provide  adequate,  accurate,  and  consistent  simulated  representations  of  the  models 
that  addresses  the  mission  objectives.  [15]  In  addition,  V&V  information  and  data  should 
be  documented,  readily  available,  and  accessible  to  DoD  users.  Therefore,  if  resources 
allow,  DoD  components  must  establish  an  M&S  VV&A  repository  that  identifies  existing 
M&S  VV&A  documentation  and  procedures  that  would  allow  DoD  users  to  identify  and 
access  M&S  VV&A  documentation,  information,  and  data.  [15] 

5.  Recommended  Practices  Guide  (RPG) 

DoD  and  military  services  have  recognized  the  growing  significance  of  modeling 
and  simulation  for  many  aspects  of  their  operations,  and  have  prepared  directives  and 
guidelines  to  provide  general  instructions  on  how,  when,  and  under  what  circumstances 
formal  VV&A  procedures  should  be  employed.  [16]  The  Recommended  Practices  Guide 
(RPG)  is  intended  to  facilitate  the  application  of  those  directives  and  guidelines,  and  to 
promote  the  effective  application  of  VV&A.  [16] 
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The  RPG  describes  the  interrelated  processes  that  make  up  VV&A  from  a  number 
of  perspectives,  as  well  as  describes  the  different  roles  and  responsibilities  of  the  various 
participants,  the  special  topics  associated  with  VV&A,  and  identify  tools  and  techniques 
that  are  used  in  VV&A.  [16]  The  RPG  answers  two  key  questions: 


a.  Why  is  VV&A  Performed? 

The  RPG  argues  that,  in  order  to  determine  whether  a  model  or  simulation 
or  federation  of  models  should  be  used  in  a  given  situation,  its  credibility  (Should  it  be 
trusted?)  must  be  established  by  evaluating  its  fitness  for  the  intended  use.  [Ref  198] 
Verification  (i.e.,  Did  I  build  the  thing  right?),  validation  (i.e.,  Did  I  build  the  right 
thing?),  and  accreditation  (i.e.,  Should  it  be  used?)  are  three  interrelated  but  distinct 
processes  that  collect  and  evaluate  evidence  to  determine  the  simulation’s  capabilities, 
limitations,  and  perfonnance  relative  to  the  real-world  systems  it  would  simulate.  [16] 
The  decision  as  to  whether  or  not  to  use  M&S  will  depend  on  the  model’s  capabilities 
and  correctness,  the  accuracy  of  its  results,  and  its  usability  within  the  specified 
application.  [16] 

VV&A  processes  are  perfonned  to  establish  the  credibility  of  the  M&S; 
credibility  depends  on  the  model’s  capability  and  accuracy  necessary  for  its  intended  use. 
Thus,  the  decision  on  whether  or  not  the  M&S  provides  the  necessary  degree  of  accuracy 
depends  upon  the  inherent  characteristics  of  the  M&S,  how  it  will  be  used,  and  the 
significance  of  the  decisions  that  may  be  reached  from  the  M&S  outputs.  [16] 

As  there  are  several  factors  that  need  to  be  considered  before  determining 
whether  a  simulation  might  be  used,  it  is  clear  that  no  simple  “yes/no”  decision  applies  in 
all  circumstances.  [16]  Therefore,  though  an  M&S  is  judged  suitable  for  one  purpose  in 
an  organization,  it  does  not  automatically  guarantee  that  it  would  be  suitable  for  the  same 
type  of  use  in  another  organization.  Similarly,  it  is  also  possible  that  the  same  M&S 
might  not  be  suitable  for  another  type  of  use  within  that  same  organization.  [16] 
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b.  Why  is  VV&A  Important? 

The  RPG  also  argues  that  the  importance  of  VV&A  is  derived  from  the 
intended  use  of  the  M&S  to  which  it  will  be  applied.  [16]  Therefore,  if  an  M&S  is  to  be 
used  to  support  training  requirements,  then  the  importance  of  VV&A  would  depend  on 
the  importance  of  the  activity  for  which  the  training  is  being  conducted,  the  degree  of 
accuracy  required  for  the  training  to  be  effective,  and  the  expected  degree  of  difficulty  for 
the  M&S  developer  in  achieving  that  accuracy.  [16] 

Continued  reliance  on  M&S  within  the  acquisition  cycle  process  increases 
the  financial  and  safety  risks  from  erroneous  or  inaccurate  M&S  results.  [16]  The 
availability  of  definitive  V&V  records  would  help  technical  managers  decide  whether  or 
not  to  use  M&S,  or  to  modify  and  re-use  an  existing  M&S,  rather  than  undertake 
development  of  a  new  simulation.  [16]  A  well-defined  V&V  process  increases  the 
potential  for  cost  savings  from  M&S  re-use.  [16]  DoD  is  encouraged  by  these  “factors” 
and  is  increasing  its  emphasis  on  employing  VV&A.  [16] 

Special  VV&A  considerations  apply  when  an  M&S  may  be  used  as  a 
substitute  for  some  prototype  field  testing  or  live  fire  testing,  because  the  extent  of  the 
VV&A  will  depend  on  the  significance  of  the  live  test  that  would  be  replaced  by  a 
simulated  test.  [16]  There  have  been  cases  throughout  history  that  illustrate  what  can  go 
wrong  when  inadequate  testing  is  performed.  [16]  This  is  where  M&S  can  help,  by 
identifying  the  essential  areas  for  testing,  and  by  prioritizing  the  required  testing 
resources.  [16] 

Finally,  the  RPG  allows  that  V&V  must  be  tailored  to  match  the  nature  of 
the  problem,  which  includes  not  only  the  situation(s)  being  simulated,  but  also  the  types 
of  decisions  that  are  driving  the  use  of  the  M&S.  [16]  Every  situation  will  be  somewhat 
different  from  the  ones  that  went  before  it  -  even  if  the  type  of  M&S  is  the  same,  so  no 
“cookbook”  VV&A  process  can  fit  all  situations  all  the  time.  Therefore,  tailoring  the 
V&V  effort  is  a  key  part  of  the  V&V  process  in  itself.  [16] 

As  mentioned  earlier,  VV&A  is  the  combination  of  three  distinct 
processes,  which  are  Verification,  Validation  and  Accreditation.  Per  DoD  Instruction 
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(DoDI)  5000.61  (as  recounted  by  the  RPG),  the  definitions  for  these  processes  are  as 
follows: 


a.  Verification 

The  process  of  detennining  that  a  model  implementation  and  its 
associated  data  accurately  represent  the  developer’s  conceptual 
description  and  specifications.  In  other  words,  Was  the  thing  built 
right?  [16] 

b.  Validation 

The  process  of  detennining  the  degree  to  which  a  model  and  its 
associated  data  provide  an  accurate  representation  of  the  real  world 
from  the  perspective  of  the  intended  uses  of  the  model.  This  could 
also  be  stated  as,  Was  the  right  thing  built?  [16] 

c.  Accreditation 

The  official  certification  that  a  model,  simulation,  or  federation  of 
models  and  simulations  and  its  associated  data  is  acceptable  for  use 
for  a  specific  purpose.  Simply  put,  Should  it  be  used?  [16] 


The  RPG  reminds  VV&A  stakeholders  that  the  accreditation  process  is  the 
official  certification  that  an  M&S  and  its  associated  data  are  fit  for  use  in  a  specific 
application.  [16]  While  accreditation  is  perceived  as  occurring  at  the  end  of  the 
development  process,  the  RPG  argues  that,  in  actuality,  this  assessment  process  should 
begin  as  early  as  possible  so  V&V  and  testing  activities  can  provide  the  appropriate  and 
sufficient  information  to  support  the  accreditation  decision,  which  is  essentially  the  user’s 
belief  in  the  credibility  of  the  M&S.  [16]  As  outlined  in  the  RPG,  the  accreditation 
process  is  comprised  of  the  following  activities: 


a.  Developing  an  Accreditation  Plan 

Develop  an  accreditation  plan  to  identify  all  the  information  needed  to 
perform  the  accreditation  assessment,  schedules,  resources,  etc.,  to  be  used  in  the 
accreditation  assessment.  [16] 
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b.  Collecting  and  Evaluating  the  Accreditation  Information 

Collect  and  evaluate  the  accreditation  information  from  the  V&V  efforts 
and  other  sources  for  assessment,  to  determine  its  completeness.  [16] 


c.  Performing  an  Accreditation  Assessment 

Perform  an  accreditation  assessment  to  determine  the  fitness  of  the 
simulation,  through  the  evidence  collected  from  the  V&V  effort  and  other  sources,  and 
prepare  an  accreditation  report  and  recommendations  for  the  user.  [16] 

Once  the  V&V  effort  and  the  accreditation  assessment  are  completed,  the 
data  collected  will  help  identify  the  risks  associated  with  using  the  simulation.  The  user 
then  weighs  the  risks  against  the  evidence  of  the  simulation  capabilities.  [16]  When 
weighing  the  risks,  the  RPG  holds  that  five  different  options  can  be  considered: 


a.  Option  1 

Full  accreditation,  or  using  the  simulation  “as  is,”  which  means  accepting 

the  risks.  [16] 


b.  Option  2 

Limited  accreditation,  or  constraining  the  application  to  minimize  the 

risks.  [16] 


c.  Option  3 

Modifying  the  simulation  as  needed  to  make  corrections  that  would  reduce 
the  risk.  This  effort  would  increase  costs  and  cause  delays.  [16] 

d.  Option  4 

Requiring  additional  infonnation,  as  needed,  to  better  understand  the  risks 
involved  and  to  build  confidence  in  the  simulation’s  fitness  before  making  a  decision. 
[16] 
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e.  Option  5 

No  accreditation  decision,  because  it  was  detennined  that  both  the  risks 
and  costs  involved  in  using  the  simulation  are  too  great.  When  no  accreditation  is 
determined,  the  user  must  select  a  different  method  to  solve  the  problem.  [16] 

The  RPG  notes  that  proper  execution  of  a  VV&A  process  involves  participants  in 
a  number  of  different  roles,  who  are  identified  in  DoDI  5000.61.  These  “key  players” 
and  their  responsibilities  are  defined  as  follows: 


a.  User 

Represent  the  organization,  group,  or  person  responsible  for  the 
overall  application.  The  user  needs  to  solve  a  problem  or  make  a 
decision  and  wants  to  use  simulation  to  do  so.  The  user  defines  the 
requirements,  establishes  the  criteria  by  which  simulation  fitness 
will  be  assessed,  determines  what  method  or  methods  to  use, 
makes  the  accreditation  decision,  and  ultimately  accepts  the 
results.  [16] 


b.  Modeling  and  Simulation  Program  Manager  (M&S  PM) 
M&S  PM  is  the  term  used  to  define  the  role  responsible  for 
planning  and  managing  resources  for  simulation  development  or 
modification,  directing  the  overall  simulation  effort,  and 
overseeing  configuration  management  and  maintenance  of  the 
simulation.  [16] 


c.  Developer 

The  role  responsible  for  actually  constructing  or  modifying  the 
simulation,  preparing  the  data  for  use  in  the  simulation,  and 
providing  technical  expertise  regarding  simulation  capabilities  as 
needed  by  the  other  roles.  [16] 


d.  Verification  and  Validation  Agent  (V&  V Agent) 

The  V&V  Agent  is  responsible  for  providing  evidence  of  the 
simulation’s  fitness  for  the  intended  use  by  ensuring  that  all  V&V 
tasks  are  properly  carried  out.  [16] 


e.  Accreditation  Agent 
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Provides  guidance  to  the  V&V  Agent  to  ensure  that  all  the 
necessary  evidence  regarding  simulation  fitness  for  use  is  obtained; 
collects  and  assesses  the  evidence;  and  provides  the  results  to  the 
User.  The  Accreditation  Authority  is  the  role  with  the 
responsibility  of  making  the  accreditation  decision.  [16] 


The  RPG  concludes  that,  in  order  for  VV&A  to  be  successful,  all  the 
above  players  must  establish  and  maintain  a  healthy  working  relationship.  [16] 


C.  COMOPTEVFORINST  5000.1A 

The  Commander,  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR) 
issued  his  latest  policy  on  VV&A  of  M&S  used  for  OT  through  COMOPTEVFOR 
Instruction  (COMOPTEVFORINST)  5000.1  A  of  September  9,  2004. 

1.  Purpose 

The  purpose  of  COMOPTEVFORINST  5000.1  A  is  “to  provide  policy  and 
process  for  COMOPTEVFOR  to  increase  the  potential  benefit  of  M&S”  in  “the  early 
planning,  organizing,  and  execution  of  credible  M&S  programs”  in  support  of 
Operational  Testing  (OT),  in  accordance  with  SECNAVINST  5200.40  &  DoDI  5000.61. 
[17] 

2.  Applicability  and  Approach 

This  policy  is  applicable  to  all  test  assets,  test  planning  aids,  and 
post-test  analysis  tools  that  substitute  for  the  varying 
characteristics  of  the  system  under  test,  supporting  forces,  the 
threat,  other  forces,  or  the  battlespace  environment  for  an 
operational  test  event  or  any  event  that  is  used  to  support 
operational  test  and  evaluation.  [17] 

The  COMOPTEVFOR  instruction  goes  on  to  state  that,  since  OT  is  not  conducted 
in  real  environments  against  real  threats  operated  by  hostile  forces,  it  is  therefore 
important  that  the  appropriate  facets  of  the  real-world  test  assets  (being  replaced  by 
M&S)  are  represented  well  enough  to  “generate  the  combat  interactions  that  are  being 
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monitored  for  the  evaluation  of  combat  effectiveness”  of  the  system  under  test  (SUT). 
[17] 

For  M&S  to  play  a  role  in  OT,  it  must  be  credible  with  the  tester 
and  stand  up  to  oversight.  Because  virtual  test  assets  are  not  as 
easy  for  testers  to  intuitively  assess  for  appropriateness  as  physical 
test  assets,  it  is  imperative  that  a  well  documented,  understandable 
procedure  is  in  place  to  characterize  the  use  of  modeled  or 
simulated  data  in  OT.  [17] 


COMOPTEVFORINST  5000.1  A  employs  a  well-defined  approach  for  M&S 

development  that  facilitates  the  accreditation  process  and  is  in  keeping  with  policies  that 

are  advocated  by  OSD  and  SECNAV.  [17] 

Accreditation  is  supported  by  the  specification  of  the  verity  and 
validity  required  for  the  applications  to  serve  as  test  assets,  and  the 
planning  necessary  to  build,  configuration  manage,  characterize, 
and  assess  the  utility  of  the  applications.  [17] 


3.  Responsibilities 

Included  in  the  stated  objectives  of  COMOPTEVFORINST  5000.1  A  is  the  need 
to  cover  all  M&S  used  to  supplement  OT  or  to  produce  DT  data  that  is  used  as  entry  to 
OT  criteria.  Consequently,  there  are  a  number  of  responsibilities  assigned  to  individuals 
to  perform  and  support  those  objectives  and  their  tasks.  [17]  The  individuals  identified  by 
the  instruction  as  needed  to  support  the  various  tasks  are  as  follows: 


a.  Commander,  Operational  Test  and  Evaluation  Force 

(1)  Provide  policy  and  guidance  for  the  intelligent  application  of 
M&S  in  OT.  (2)  Accredit  the  use  of  M&S  outputs  to  resolve 
critical  operational  issues  (COI)  during  operational  evaluation 
(OPEVAL)  or  follow-on  operational  test  and  evaluation  (FOT&E). 
[17] 


b.  Assistant  Chiefs  of  Staff,  Warfare  Divisions  (ACOS(W)) 

(1)  Provide  an  assessment  to  COMOPTEVFOR  of  impacts  to  the 
test  program  of  using  M&S  data  to  resolve  COI  during  OPEVAL 
or  FOT&E.  (2)  Accredit  the  use  of  M&S  data  to  assess  COI  used 
in  tests  other  than  OPEVAL  and  FOT&E.  [17] 
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c.  Operational  Test  Coordinators  (OTC) 

(1)  Assess  whether  appropriate  program  resources  have  been 
allocated  and  applied  to  M&S  application  development, 
management,  and  use  to  mitigate  the  risk  of  using  M&S  data  in  the 
OT  program.  (2)  Make  recommendations  to  the  ACOS(W)  on  the 
program  viability  of  M&S  applications  for  use  in  OT.  [17] 


d.  Operational  Test  Directors  (OTD) 

(1)  Manage  the  use  of  M&S  as  a  test  asset  for  the  conduct  of  OT. 

(2)  Make  recommendations  to  the  ACOS(W)  on  the  appropriate¬ 
ness  of  M&S  applications  to  support  the  resolution  and  assessment 
ofCOIinOT.  [17] 


e.  Modeling  and  Simulation  Manager  (MSM) 

(1)  Manage  the  use  of  M&S  applications  to  support  OT  across  the 
command.  (2)  Provide  direct  support  to  the  ACOS(W),  OTC,  and 
OTD  in  executing  the  policy,  responsibilities,  processes  relating  to 
the  use  of  M&S  in  OT.  (3)  Review  M&S  program  data  requested 
from  PM  for  rigor  and  relevance  to  the  application’s  intended  use 
within  OPTEVFOR.  (4)  Make  recommendations  to  the  ACOS(W) 
on  the  technical  soundness  of  M&S  applications  with  respect  to 
programmatics  and  appropriateness  for  use  to  support  OT.  (5) 
Maintain  the  corporate  knowledge  on  the  use  of  M&S  to  support 
OT  and  apply  that  knowledge  to  improving  the  policies  and 
processes  at  OPTEVFOR.  (6)  Work  with  the  command’s  Staff 
Editor  to  maintain  and  update  formats  for  M&S  process 
documentation  in  the  command’s  OT&E  reference  library 
directory,  OT&E  formats  folder  and  to  update  and  maintain 
detailed  instructions  for  completing  the  fonnats  in  the  Modeling 
and  Simulation  folder.  [17] 

4.  Accreditation 

The  policy  section  of  COMOPTEVFORINST  5000.1  A  discusses  the  accreditation 
approval  process.  Although  the  officers  and  staff  of  COMOPTEVFOR  do  not  lead  M&S 
efforts,  they  participate  by  generating  OT  acceptability  requirements  and  providing 
operational  insight  at  both  technical  and  management  reviews.  [17]  The  command 
establishes  and  assesses  the  accreditation  criteria  for  all  M&S  that  will  be  used  in  OT, 
and  often  assists  acquisition  program  offices  in  the  development  of  other  accreditation 
criteria  for  M&S.  [17]  COMOPTEVFOR  requests  that  M&S  accreditation  decisions  be 
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scheduled  well  ahead  of  the  test  event  so  as  to  allow  alternative  approaches  for  testing  in 
the  event  accreditation  is  not  granted.  Consequently,  PMs  are  advised  to  submit 
sufficient  verification  and  validation  (V&V)  data  in  timely  increments  to  support  the 
accreditation.  [17]  The  instruction  states  that  PMs  and  COMOPTEVFOR  officials  should 
jointly  detennine  if  assets  and  resources  are  appropriate  to  support  testing.  [17]  This 
decision  should  be  made  early  within  the  acquisition  process,  along  with  identifying  the 
proper  M&S  requirements,  which  include  programmatic,  operational  and  performance 
factors.  [17] 

The  instruction  states  that  M&S  accreditation  letters  are  issued  by  the 
accreditation  authority  (i.e.,  COMOPTEVFOR)  and  may  include  an  accreditation 
assessment  report  that  confirms  that  the  M&S  is  adequate  and  credible  for  the  intended 
use.  [17]  The  level  of  demonstration  that  is  acceptable  for  the  accreditation  criteria  is 
determined  by  the  MSM  and  status  is  then  provided  to  the  PM.  [17]  COMOPTEVFOR 
establishes  the  minimum  fidelity  requirements  for  the  V&V  data  that  supports 
accreditation  for  OT,  during  the  development  of  the  V&V  plan,  including  the  acceptance 
of  the  V&V  techniques  to  be  used,  the  application  method,  and  the  degree  of  observation 
required  to  build  confidence  in  the  V&V  results  to  be  assessed  for  accreditation.  [17] 


5.  Credibility  and  Resourcing 

COMOPTEVFORINST  5000.1  A  notes  that  M&S  development  resources  should 
reflect  the  technical  expertise  needed  to  develop,  manage,  and  operate  the  VV&A 
process,  as  well  as  identify  the  financial  and  material  requirements  for  accreditation.  [17] 
These  resources  that  are  needed  to  support  DT  and  OT  uses  must  be  identified  as  early  as 
possible  to  enable  the  program  to  budget  appropriately.  [17] 

Finally,  the  instruction  points  out  that  the  OTC  will  ensure  that  adequate 
responsibilities  are  identified  in  the  test  and  evaluation  strategies  (TES)  and  adequate 
resources  are  allocated  in  the  Test  and  Evaluation  Master  Plan  (TEMP)  to  ensure  that 
M&S  applications  produce  credible  data  for  use  in  OT  (and  DT  events  that  support  OT). 
[17] 
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D.  VV&A  PROCESSES  OF  U.S.  NAVY  SURFACE  SHIPS 

This  section  will  address  the  VV&A  processes  of  the  LPD-17,  DDG-51,  and  DD- 
X  (DD-21)  U.S.  Navy  surface  ships  warfare  programs,  which  were  previously  described 
in  Chapter  II. 

1.  LPD-17  VV&A  Process 

The  LPD-17  M&S  program  is  based  upon  the  requirements  and  fundamentals 
described  in  the  DoD/DoN  5000  series  of  directives  and  instructions  and  in  the  DMSO- 
produced  VV&A  RPG.  [10] 

The  LPD-17  VV&A  Guide  is  tailored  from  the  DMSO  VV&A  RPG  and  is  used 
by  the  LPD-17  team  members  to  define  and  create  the  VV&A  activities  and 
documentation  required  for  each  M&S  activity.  [10]  The  VV&A  Guide  assists  team 
members  as  they  establish  the  level  of  credibility  for  their  specific  M&S  effort  by 
providing  both  a  process  framework  and  a  set  of  minimum  requirements  for  each  M&S 
activity  family.  [10]  These  requirements  take  a  “reasonable”  approach  to  establishing  an 
M&S  application’s  credibility,  with  the  ultimate  goal  of  obtaining  well  defined  and 
understood  M&S  requirements.  This  also  includes  the  goals  that: 

a.  the  capabilities,  limitations,  assumptions,  and  approximations  of 
the  application  are  documented  and  understood; 

b.  the  performance  of  the  selected  application  meets  the  M&S 
requirements  of  the  task;  and 

c.  the  input  data  used  is  correct  and  sufficiently  accurate.  [10] 

2.  DDG-51  W&A  Process 

The  PMS  400  Flight  IIA  LFT&E  accreditation  approach  requires  the  technical 
review  panel  (TRP)  gather  data  using  an  accreditation  questionnaire  and  conduct  reviews 
with  M&S  proponents  that  discuss  model  descriptions;  measures  of  effectiveness  and 
performance;  variables  calculated;  objects  and  functions  modeled  and  their  limitations; 
key  sources  of  input  data;  configuration  management;  documentation  status,  V&V 
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techniques  used  and  specific  V&V  that  were  conducted;  and  capabilities,  limitations,  and 
limiting  assumptions.  [12]  PMS  400 ’s  Simulation  Control  Panel  (SCP)  conducts  the 
analysis  of  capabilities,  limitations,  and  limiting  assumptions  that  apply  to  resolving  the 
LFT&E  issues.  Their  Accreditation  Review  Panel  (ARP)  assesses  the  risks  associated 
with  M&S  weaknesses  and  makes  an  accreditation  recommendation  to  the  Accreditation 
Authority  (AA),  who  is  PMS  400  in  for  the  M&S  used  for  OT.  [12] 

Some  of  the  lessons  learned  from  the  Flight  IIA  Accreditation  Process  are: 

a.  Model  documentation  may  not  be  readily  available  for  legacy 

M&S.  [12] 

b.  Funding  was  not  initially  provided  to  document  the  Configuration 
Management  process.  [12] 

c.  V&V  terminology  and  expectations  needed  to  be  better  defined 
and  managed.  Questions,  such  as  What  is  V&V?  and  “If  M&S  is  accredited,  does  the 
process  have  to  be  repeated?”  need  clearer  answers.  [12] 

d.  Not  clear  whether  the  results  of  previous  V&V  efforts  were 
documented.  [12] 

e.  Need  to  better  understand  how  the  capabilities  and  limitations  of 
the  M&S  will  affect  answering  the  critical  issues  before  deciding  whether  or  not  to  use  a 
particular  M&S.  [12] 

f.  Some  LFT&E  issues  cannot  be  answered  through  M&S  alone.  [12] 

g.  M&S  is  crucial  to  conducting  a  thorough  evaluation  of  ship 
survivability.  [12] 

3.  DD-X  (DD-21)  VV&A  Process 

PMS  500  has  defined  an  accreditation  process  to  ensure  that  the  program  office 
has  acquired  sufficient  insight  and  confidence  in  relevant  M&S  to  support  the  DD-X 
program.  [18]  This  process  is  structured  into  four  parts  to  illustrate  how  it  can  be  tailored 
to  a  particular  development  phase.  [18] 
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a.  The  first  part  is  risk/resource  management,  where  techniques  are 
used  to  optimize  resource  allocation  for  accreditation  efforts  and  M&S  are  selected  and 
submitted  for  accreditation  (after  meeting  PMS  500  defined  criteria  on  prioritizing  the 
M&S  of  interest).  [18]  A  Simulation  Control  Panel  (SCP)  conducts  an  initial, 
engineering-level  review  to  ensure  that  the  criteria  have  been  met,  and  then  notifies  the 
M&S  proponent,  ARP,  and  AA  of  the  schedule  for  V&V  review.  [18]  When  notified,  the 
M&S  configuration  manager  begins  tracking  model  status.  During  this  phase,  the  AA 
may  delegate  a  functional  area  leader  as  accreditation  authority  to  determine  the  risk 
associated  with  the  M&S  used.  [18] 

b.  The  second  part  is  the  insight/V&V  analysis.  This  is  when  the 
SCP  reviews  the  proponent-proposed  accreditation  success  criteria  and  assesses  the  V&V 
information  that  would  include  V&V  plans,  reports,  and  observations  associated  with 
each  phase  of  the  M&S  development.  [18] 

c.  The  third  part  is  judgment.  This  is  when  the  SCP  informs  the  ARP 
of  the  assessments  and  forwards  a  recommendation  of  accreditation  that  is  consistent  with 

the  development  schedule.  The  ARP  then  provides  further  review  and  solicits  approval 
from  the  accreditation  authority.  [18]  Accredited  M&S  are  logged  in  a  repository  for 
possible  reuse.  [18] 

d.  The  fourth  part  is  when  the  V&V  review  results  are  given  as 
feedback  to  the  M&S  proponent.  The  feedback  would  include  information  such  as 
clarifications,  blunders,  long-lead  material,  and  PM  approval.  [18]  This  feedback  is 
controlled  by  phase-dependent  policies;  for  instance,  phase  II  assessments  of  V&V 
results  are  retained  by  PMS  500  until  a  contractual  down-selection  has  been  made.  This 
helps  avoid  any  government  influence  on  the  competition.  [18] 

4.  Conclusion  of  Review  of  Surface  Ship  Program  VV&A  Processes 

The  described  processes  indicate  that  some  surface  ship  acquisition  program 
managers  have  adopted  VV&A  practices  that  are  similar  to  what 
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COMOPTEVFORINST.5000.1A  prescribes  for  OT  and  are  in  line  with  the  policies 
promulgated  by  OSD  and  SECNAV.  However,  from  review  of  other  processes,  it 
appears  that  program  managers,  in  general,  still  may  not  realize  the  full  benefits  (with 
respect  to  cost  reduction  and  risk  avoidance)  that  could  be  achieved  by  closely  following 
the  standards  of  the  COMOPTEVFORINST  5000.1  A  VV&A  process  to  support  their 
M&S  requirements  for  OT. 


E.  CONCLUSION 

This  chapter  described  DoD’s  VV&A  processes,  as  well  as  the  U.S.  Navy’s 
COMOPTEVFORINST  5000.1  A  VV&A  process  and  the  VV&A  processes  of  three 
surface  ship  programs.  The  next  chapter  will  describe  the  research  methodology  used  to 
collect  data  for  this  thesis. 
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IV.  RESEARCH  METHODOLOGY 


A.  INTRODUCTION 

This  chapter  describes  the  research  methodology  that  was  used  to  obtain  the  data 
for  this  thesis.  The  first  step  taken  was  the  identification,  by  category,  of  the  major 
stakeholders  who  participate  in  the  VV&A  of  M&S  used  for  the  OT  of  newly  acquired  or 
updated  U.S.  Navy  surface  ships.  Then,  information  and  data  on  the  practices  and 
expectations  of  VV&A  were  obtained  through  literature  reviews,  forum  attendances,  and 
semi-structured  interviews  of  available  stakeholders.  Data  from  the  semi-structured 
interviews  was  analyzed  and  reported  for  each  of  ten  interview  questions  by  W&A 
stakeholder  category  and  stakeholder  management  level.  Conclusions  were  developed  by 
a  systematic  review  of  the  results  against  the  thesis  objectives  and  the  primary  and 
subsidiary  research  questions.  Recommendations  were  fonned  through  consideration  of 
the  conclusions  and  the  researchers’  general  experience  and  expectations  for  the  future  of 
VV&A. 

B.  IDENTIFICATION  OF  MAJOR  STAKEHOLDERS 

It  became  clear  at  the  onset  that  a  key  factor  for  analyzing  this  essential  sector  of 
the  U.S.  Navy’s  VV&A  business  was  the  identification  and  categorization  of  the  process 
stakeholders.  By  definition,  these  stakeholders  “own  a  stake”  in  the  VV&A  process, 
whether  through  direct  investment  of  their  resources  and  the  resulting  expectation  of  a 
return  on  investment,  or  because  they  are  responsible  for  ensuring  that  the  models  and 
simulations  developed  are  credible  for  their  intended  use.  Four  distinct,  fundamental 
roles  are  played  by  U.S.  Navy  stakeholders  that  explain  their  individual  practices  and 
shape  their  various  expectations  for  the  outputs  and  outcomes  of  the  VV&A  process  used 
for  OT. 


1.  Tester/Accreditor 

The  first  of  the  fundamental  stakeholder  categories  that  participate  in  the  VV&A 
of  M&S  used  for  OT  of  U.S.  Navy  surface  ships  is  that  of  the  Tester/Accreditor.  This  is 
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a  single  stakeholder  category  that  nevertheless  encompasses  a  dual  function:  The 
stakeholder,  in  order  to  establish  the  suitability  and  effectiveness  military  items,  either 
conducts  physical  testing  of  hardware  in  real  world  enviromnents  or  accredits  models  and 
simulations  used  in  virtual  environments.3  Making  the  operational  tester  also  be  the 
accreditor  of  the  M&S  employed  for  OT  is  essential:  By  maintaining  both  functions  at 
one  activity,  the  command  has  the  complete  authority  it  needs  to  officially  state  that  a 
warfighting  system’s  suitability  and  effectiveness  has  been  demonstrated.  For  U.S.  Navy 
OT,  the  Tester/Accreditor  stake-holder  role  is  fulfilled  by  the  Operational  Test  Directors 
(OTDs)  at  COMOPTEVFOR,  as  they  are  solely  responsible  for  the  operational  test  and 
evaluation  of  all  U.S.  Navy  warfighting  systems,  including  surface  ships,  and  the 
accreditation  of  all  M&S  used  in  support  of  their  OT.  Of  further  relevance, 
COMOPTEVFOR  is  organizationally  independent  of  the  other  VV&A  stakeholder 
groups  within  the  U.S.  Navy.  Thus,  the  accrediting  role  played  by  COMOPTEVFOR 
(through  the  OTDs)  in  this  VV&A  process  is  free  of  those  interests  that  constrain  the 
other  stakeholders. 

2.  Resource  Sponsor 

The  second  stakeholder  group  in  this  VV&A  process  is  that  of  Resource  Sponsor. 
(Other  titles  frequently  used  in  U.S  military  organizations  for  this  group  are 
“Requirements  Officer”  and  “Combat  Developer.”)  The  Resource  Sponsor  identifies  the 
operational  requirement  and/or  capability  desired  for  the  warfighting  system  under  his/ 
her  cognizance  and  obtains  the  funding  needed  to  acquire  the  system.  Regarding  the 
subject  at  hand,  the  Resource  Sponsor  must  see  that  adequate  funding  and  time  are 
budgeted  for  the  conduct  of  the  VV&A  and  agree  that  developing  M&S  products  will 
best  demonstrate  the  satisfaction  of  requirements  and/or  the  fulfillment  of  warfighting 
capabilities.  For  the  U.S.  Navy,  the  staff  codes  responsible  for  overseeing  particular 


3  The  same  person,  i.e.,  the  Operational  Test  Director  (OTD)  under  COMOPTEVFOR,  is  specifically 
responsible  to  COMOPTEVFOR  for  ensuring  that  both  physical  tests  and  models/simulations  address  the 
COIs  for  the  OT  of  a  particular  U.S.  Navy  surface  ship  (and/or  its  weapons  systems).  A  single  focal  point 
ensures  that  the  COIs  are  sufficiently  and  completely  addressed.  The  OTD  is  assisted  by  the  COMOPTEV¬ 
FOR  VV&A  Manager  in  reviewing  and  recommending  M&S  used  for  OT  to  their  boss,  COMOPTEVFOR. 
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warfighting  platforms  within  the  Chief  of  Naval  Operations  (OPNAV)  play  this 
stakeholder  role.  In  the  case  of  surface  ships,  Code  N76  is  the  Resource  Sponsor. 
Because  of  his/her  oversight  function  in  the  acquisition  of  surface  ships,  the  N76 
Resource  Sponsor  may  often  play  a  mediating  role  with  respect  to  the  other  W&A 
stakeholders. 


3.  Program  Manager 

The  third  stakeholder  group  in  this  VV&A  process  is  that  of  Program  Manager 
(PM).  The  PM  is  the  senior  manager  in  the  acquisition  office  that  charters  the 
development  of  the  models  and  simulations  and  proposes  that  they  be  used  (instead  of 
testing  with  physical  assets)  to  demonstrate  satisfaction  of  OT  requirements.  Thus,  the 
PM  is  also  known  as  the  M&S  proponent  in  the  VV&A  for  OT  process.  The  PM, 
through  his/her  program  office,  is  responsible  for  delivering  a  warfighting  system  within 
budget  and  schedule  that  meets  prescribed  requirements,  and  is  thereby  granted  a  certain 
amount  of  discretion  within  these  constraints  to  accomplish  the  program’s  mission. 
Therefore,  he/she  may  consider  a  number  of  options,  including  whether  to  propose 
physical  or  virtual  testing  for  the  fulfillment  of  OT.  At  the  same  time,  he/she  must 
ultimately  convince  the  U.S.  Navy  surface  ship  acceptance  authorities  to  accept  the 
program’s  deliverables,  whether  they  be  demonstrated  by  physical  or  virtual  testing. 
Thus,  the  PM  acts  as  the  prime  mover  in  the  VV&A  process  by  proposing  action  to  the 
other  stakeholders. 

4.  VV&A  Standards  Official 

The  fourth  player  in  the  VV&A  of  M&S  used  for  OT  of  U.S.  Navy  surface  ships 
is  that  of  the  “VV&A  Standards  Official”  category.  This  group  is  interested  in  seeing 
that  all  of  the  VV&A  stakeholders  are  following  a  standard  process  that  is  documented 
and  repeatable.  The  Navy  Modeling  and  Simulation  Office  (NMSO  -  fonnerly 
NAVMSMO  -  Navy  Modeling  and  Simulation  Management  Office)  perfonns  this 
function  within  the  U.S.  Navy.  Though  a  part  of  the  U.S.  Navy’s  OPNAV  organization, 
this  group  operates  independently  of  the  Resource  Sponsors  (as  well  as  the  other 
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stakeholder  groups),  but  supports  all  of  the  other  stakeholders  in  communicating  and 
encouraging  the  use  of  standards  in  VV&A.  Figure  1  depicts  the  basic  relationships 
between  the  four  stakeholder  groups  and  shows  (as  depicted  by  the  dotted  line)  that  the 
VV&A  Standards  Official  group  informs  the  other  three. 


Figure  1.  Relationships  between  OT  VV&A  Stakeholder  Groups  within  U.S.  Navy 

C.  LITERATURE  REVIEWS 

Research  began  by  identifying  the  published  sources  of  information  and  data  on 
the  practices  and  expectations  of  VV&A  for  M&S  used  to  satisfy  OT  requirements  of 
U.S.  Navy  surface  ships  (and  ship  weapons)  that  are  completing  their  acquisition  process. 
Published  information  was  located  on  the  official  websites  of  the  U.S.  Navy  and 
Department  of  Defense  (DoD)  stakeholders  and  downloaded.  On  several  occasions,  it 
was  more  expedient  to  directly  request  that  the  stakeholders  provide  the  documents  as  e- 
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mail  attachments  or  hardcopies  delivered  in  person  at  forums.  Since  COMOPTEVFOR 
accredits  the  models  and  simulations  used  to  satisfy  the  requirements  of  operational  test 
and  evaluation,  it  was  paramount  to  begin  the  relevant  literature  reviews  at  this 
command.  Paragraphs  1  and  2,  below,  identify  the  key  documents.  OPNAV,  the  Navy 
Secretariat  (SECNAV),  and  DoD  publish  documents  that  provide  overarching  VV&A 
policy  and  context  for  the  policy  issued  by  COMOPTEVFOR.  Paragraph  3,  below, 
discusses  this  area  of  the  literature  review.  Next,  a  review  was  conducted  of  the  VV&A 
and  M&S  process  plans  released  by  U.S.  Navy  surface  ship  acquisition  programs.  These 
process  plans  were  either  published  on  the  programs’  websites  or  released  to  the 
researchers  by  the  program  offices.  Finally,  the  researchers  obtained  important  insights 
on  the  VV&A  process  through  review  of  related  professional  publications,  many  of 
which  were  authored  by  the  stakeholders  that  participate  in  the  VV&A  of  U.S.  Navy 
surface  ship  M&S.  Paragraph  5,  below,  covers  this  review.  In  all,  five  areas  of  literature 
review  were  conducted  to  obtain  the  relevant  information. 

1.  Review  of  COMOPTEVFORINST  5000.1A,  dated  SEP  9  2004 

A  review  was  conducted  of  the  recently  updated  COMOPTEVFOR  Instruction 
5000.1  A  of  9  September  2004,  with  the  subject,  “Use  of  Modeling  and  Simulation 
(M&S)  in  Operational  Test  (OT).”  [17]  Chapter  III  of  this  thesis  provides  a  detailed 
description  of  this  instruction.  The  Instruction  cancels  the  previous  instruction  (dated  5 
September  1995),  which  is  described  below. 

2.  Review  of  COMOPTEVFORINST  5000.1,  dated  SEP  5  1995 

To  appreciate  what  was  changed  by  the  update,  a  review  was  conducted  of  the 
superseded  instruction,  COMOPTEVFOR  Instruction  5000.1  of  5  September  1995,  and 
its  supplementary  instructions,  COMOPTEVFOR  Memorandum  3980  Ser  00T/312  of  29 
April  1998  and  COMOPTEVFOR  Policy  and  Information  Notice  99-01  of  13  May  1999. 
Most  of  the  experience  of  the  stakeholders  interviewed  was  in  compliance  with  this 
instruction  and  its  two  supplements.  While  the  title  of  this  earlier  instruction  is 
essentially  identical  to  its  replacement,  the  stated  purpose  of  COMOPTEVFOR 
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Instruction  5000.1  of  5  September  1995  is  more  lengthy  and  detailed.  This  instruction 
was  issued  at  the  time  to  provide  interim  policy  and  guidance,  with  the  expectation  that  it 
would  be  modified  when  “...  subordinate  instructions  are  published  ....”  [19] 
(Furthermore,  one  of  the  three  references  cited  was  still  in  draft  form.)  The  instruction 
included  six  enclosures  (which  made  the  whole  document  28  pages  long)  that  provided  a 
glossary  of  terms  and  a  range  of  sample  documents  for  an  acceptance  criteria  matrix,  a 
model  management  plan,  a  memorandum  of  agreement  for  Verification  and  Validation 
(V&V)  of  M&S  between  independent  agencies,  some  organizational  schematics  for 
Simulation  Management  Boards  (SMB)  and  Simulation  Control  Panels  (SCP),  and  an 
end-to-end  accreditation  process  flowchart.  Regarding  COMOPTEVFOR’s  two 
supplemental  documents:  the  29  April  1998  memorandum  discussed  the  “re¬ 
accreditation”  [20]  of  previously  accredited  M&S  and  clarified  the  Command’s  role  in 
the  accreditation  of  M&S  that  are  components  of  systems  under  test;  and  the  13  May 
1999  Policy  and  Information  Notice  provided  “guidance  in  accurately  documenting  the 
accreditation  processes  and  results  ...”  [21]  including  three  enclosures  for  (1) 
accreditation  plan  outline,  (2)  accreditation  letter  outline,  and  (3)  accreditation 
assessment  report. 

3.  Review  of  Related  U.S,  Department  of  Defense  Policy  Documents 

Reviews  were  conducted  of  other  VV&A,  M&S,  and  OT  documents  promulgated 
by  COMOPTEVFOR,  as  well  as  the  related  directives,  instructions,  guidance,  and 
information  notes  issued  by  OPNAV,  SECNAV,  DOT&E,  and  OUSD.  These  documents 
provide  overarching  policy,  guidance,  and  information  on  VV&A,  M&S,  OT,  and  the 
acquisition  of  major  defense  systems.  As  such,  they  provide  the  “support  net”  that 
enables  compliance  with  COMOPTEVFORINST  5000.1  A  of  9  September  2004. 
Chapter  III  provides  detailed  descriptions  of  two  of  the  OUSD  documents,  DoD 
Instruction  (DoDI)  5000.61  on  “DoD  Modeling  and  Simulation  (M&S)  Verification, 
Validation,  and  Accreditation  (VV&A)”  of  13  May  2003  and  the  VV&A  “Recommended 
Practices  Guide”  of  4  August  2004 
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4.  Review  of  U.S.  Navy  Surface  Ship  Acquisition  Program  Publications 

Reviews  were  conducted  of  the  VV&A  and  M&S  process  plans  released  by  U.S. 
Navy  surface  ship  acquisition  programs.  Most  of  the  program  offices  contacted  by  the 
researchers  had  developed  a  program  specific  VV&A  process  chart  that  was  used  to 
guide  program  participants  through  their  VV&A  program  and  inform  their  Resource 
Sponsors  and  COMOPTEVFOR.  These  process  plans  were  either  published  on  the 
programs’  websites  or  released  to  the  researchers  by  the  program  offices.  Chapter  III 
provides  brief  descriptions  of  three  of  the  VV&A  efforts  of  U.S.  Navy  surface  ship 
acquisition  programs. 

5.  Review  of  Related  Professional  Publications 

Reviews  were  conducted  of  related  professional  websites,  forum  briefings, 
symposium  papers,  journal  articles,  books,  theses,  and  studies.  Those  publications  that 
were  found  to  provide  additional  insight  to  the  mission  of  VV&A  for  the  U.S.  Navy  are 
listed  in  the  Bibliography.  Chapter  III  cited  a  number  of  these  publications  in  describing 
DoD  and  U.S.  Navy  VV&A  processes. 


D.  VV&A  FORUM  ATTENDANCE 

The  research  team  attended  five  VV&A  Forums  (formerly:  TWGs  -  Technical 
Working  Groups)  co-hosted  by  the  Navy  Modeling  and  Simulation  Office  (NMSO)  and 
the  U.S.  Navy  (or  other  U.S.  defense  command)  where  the  forum  was  held.  The  thesis 
advisor  attended  the  first  of  the  forums  and  introduced  the  research  team  to  the  forum. 
The  research  students  attended  four  subsequent  forums.  The  forums  attended  by  the 
research  team  were: 

1.  VV&A  Forum  #16,  on  3  March  2004,  in  San  Diego.  CA 

The  NMSO  (formerly  NAVMSMO)  VV&A  Technical  Working  Group  (TWG) 
held  a  workshop  (now  called  a  “VV&A  Forum”)  on  3  March  2004  at  the  SPAWAR 
Systems  Center  in  San  Diego,  CA.  The  purpose  of  the  forum  was  to  allow  the  U.S.  Navy 
M&S  communities  an  opportunity  to  stay  current  on  the  latest  VV&A  policy,  guidance, 
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resources,  and  tools  available.  In  addition  to  briefings  on  current  VV&A  topics,  an  open 

discussion  of  the  VV&A  communities’  needs  and  issues  was  facilitated,  in  order  to 

identify  commonalities  and  possible  solutions.  [22]  The  thesis  advisor  attended  and  gave 

a  briefing  entitled,  “Studying  the  A  in  VV&A.”  The  thesis  research  team  was  introduced 

and  the  research  motivation,  background,  research  question,  methodology,  and  future 

steps  of  the  thesis  research  were  briefed.  As  the  minutes  report,  the  presentation 

prompted  forum  discussion,  raising  topics  such  as 

. . .  the  need  to  look  at  the  larger  issues  of  M&S  re-use  given  that 
the  accreditation  authority  is  far  removed  from  the  original  M&S 
development  process  . . .  [and]  the  need  to  account  for  funding  and 
resources  regardless  of  the  status  of  the  M&S  development  status. 

...  In  the  same  vein,  questions  were  posed  on  how  to  ensure  the 
accreditation  of  new  models  given  the  difficulty  of  accrediting 
legacy  models.  The  presenter  pointed  out  the  benefits  of  peer 
review.”  [22] 

2.  VV&A  Forum  #17,  on  7  May  2004,  in  West  Bethesda,  MD. 

The  thesis  researchers  attended  the  NMSO  VV&A  Forum  on  7  May  2004  at  the 
Carderock  Division  of  the  NAVSEA  Warfare  Center  in  West  Bethesda,  MD.  VV&A 
Forum  #17  provided  the  U.S.  Navy  M&S  communities  an  opportunity  to  keep  abreast  of 
the  latest  VV&A  policy,  guidance,  resources,  and  tools  available.  In  addition  to  briefings 
on  current  VV&A  topics,  an  open  discussion  of  the  VV&A  communities’  needs  and 
issues  was  facilitated,  in  order  to  identify  commonalities  and  possible  solutions. 
COMOPTEVFOR’s  VV&A  Manager  presented  a  briefing  on  the  philosophy  of  OT  and 
the  use  of  M&S  to  represent  aspects  of  the  systems  under  test,  threat  forces,  and  the 
operating  environment.  The  presentation  also  indicated  COMOPTEVOR’s  plans  to 
update  COMOPTEVORINST  5000.1  by  releasing  COMOPTEVFORINST  5000.1  A 
within  the  year.  [23] 


3.  VV&A  Forum  #18,  on  14  September  2004,  in  Norfolk,  VA. 

The  thesis  researchers  attended  the  NMSO  VV&A  Forum  on  14  September  2004 
at  the  Commander  Operational  Test  Force  (COMOPTEVFOR)  Headquarters  in  Norfolk, 
VA.  VV&A  Forum  #18  provided  the  U.S.  Navy  M&S  communities  an  opportunity  to 
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obtain  updates  on  recent  VV&A  policy,  guidance,  resources,  and  tools.  In  addition  to 
briefings  on  current  VV&A  topics,  an  open  discussion  of  the  VV&A  communities'  needs 
and  issues  was  facilitated,  in  order  to  identify  commonalities  and  possible  solutions.  In 
addition  to  presentations  on  current  VV&A  topics,  a  round  table  discussion  followed  and 
volunteers  were  solicited  for  reviewing  the  draft  DoN  M&S  VV&A  Implementation 
Handbook,  Volume  III  (Case  Studies  and  Lessons  Learned).  COMOPTEVFOR’s  VV&A 
Manager  announced  that  COMOPTEVFORINST  5000.1  A  was  just  signed  (on  9 
September  2004)  and  distributed  hardcopies  to  forum  participants.  The  thesis  researchers 
presented  a  brief  to  the  forum  that  discussed  the  primary  research  question  and  the 
research  methodology,  and  requested  interview  participants  for  the  thesis  research.  The 
issue  of  VV&A  “practice  versus  expectations”  was  briefed  and  the  key  VV&A  policy 
documents  (for  literature  review)  and  the  major  stakeholder  categories  (formulated  by  the 
researchers)  were  identified.  [24] 

4.  VV&A  Forum  #21,  on  11-12  May  2005,  in  Dahlgren,  VA. 

The  thesis  researchers  attended  the  NMSO  VV&A  Forum  on  1 1-12  May  2005  at 
the  Dahlgren  Division  of  the  NAVSEA  Warfare  Center  in  Dahlgren,  VA.  VV&A  Forum 
#21  provided  the  U.S.  Navy  M&S  communities  an  opportunity  to  leam  the  latest 
regarding  VV&A  policy,  guidance,  resources,  and  tools  available  and  to  discuss 
difficulties  and  areas  of  concern  regarding  program-specific  VV&A  efforts.  This  forum 
focused  on  DoN  M&S  VV&A  efforts  and  M&S  requirements,  with  the  goal  of 
facilitating  greater  awareness  of  standards,  best  practices,  and  areas  of  continuing 
concern.  [25]  The  NAVSEA  Dahlgren  Accreditation  Team  (NDAT)  presented  their 
VV&A  process,  which  included  detailed  process  charts  that  were  adapted  (i.e.,  tailored) 
to  the  specific  requirements  of  the  various  surface  ship  (and  weapons)  acquisition 
programs  supported. 

5.  VV&A  Forum  #23,  on  28  Feb.  -  1  March  2006,  in  Chesapeake,  VA. 

The  thesis  researchers  attended  the  NMSO  VV&A  Forum  from  28  February  2006 
to  1  March  2006  at  the  U.S.  Joint  Forces  Command  (USJFCOM)  in  Chesapeake,  VA. 
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VV&A  Forum  #23  provided  the  U.S.  Navy,  the  other  U.S.  military  services,  and  the 
USJFCOM  M&S  communities  an  opportunity  to  gain  new  insights  on  VV&A  policy, 
guidance,  resources,  and  tools  available  and  to  discuss  difficulties  and  areas  of  concern 
regarding  program-specific  VV&A  efforts.  The  first  day  of  the  forum  concentrated  on 
DoN,  DoD,  and  Joint  M&S  VV&A  efforts  with  the  aim  of  bringing  greater  awareness  of 
standards,  best  practices,  and  areas  of  continuing  concern  in  the  areas  of  experimentation 
and  testing  and  evaluation.  The  second  day  was  dedicated  to  specific  “breakout”  sessions 
that  provided  focused  discussion  on:  (1)  Experimentation  for  M&S  and  special 
considerations  for  VV&A;  and  (2)  M&S  VV&A  in  testing  and  evaluation.  [26] 


E.  SEMI-STRUCTURED  INTERVIEWS 

Early  in  the  planning  phase  of  the  thesis,  the  researchers  developed  the  basic 
concept  of  conducting  interviews  of  stakeholders,  using  a  questionnaire,  to  develop  data 
that  would  be  useful  in  answering  the  research  questions.  In  the  semi-structured 
interview,  the  interviewer  would  pose  a  set  of  standard  questions  to  the  interviewee,  but 
would  also  invite  the  interviewee  to  comment  on  any  additional  VV&A  related  items  that 
seemed  important.  Thus,  a  mix  of  structured  data  and  free  comments  would  be  expected 
from  the  interviewees.  A  five  phase  approach  was  employed  to  develop  the  data  under 
the  semi-structured  interview  methodology. 

1.  Pilot  Questionnaire 

The  researchers  developed  a  “pilot  questionnaire”  that  was  used  to  obtain  initial 
feedback  from  a  representative  sample  of  participants  in  the  VV&A  of  M&S  used  for  the 
OT  of  newly  acquired  or  updated  U.S.  Navy  surface  ships.  This  questionnaire  targeted 
the  PM  stakeholder  group  and  was  composed  of  2 1  questions,  which  ranged  from  its  first 
question: 

“What  surface  ship  acquisition  program(s)  did  you  work  on 
or  assist  that  used  M&S  tools  (models)  to  support  OT?  ” 
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...  that  was  designed  to  capture  the  VV&A  experience  of  the  stakeholder,  to  its  last 
question: 


In  your  opinion,  to  what  extent  would  the  distinctly 
different  organizational  structures  of  U.S.  Navy  surface 
ship  acquisition  programs  found  at  NAVSEA  impact  a 
standard  implementation  of  the  VV&A  process  advocated 
by  COMOPTEVFORINST  5000. 1  ? 

...  that  was  fashioned  to  obtain  a  potential  explanation  for  not  following  the  standard 
implementation  of  COMOPTEVFORINST  5000.1.  The  researchers  expected  that 
stakeholder  feedback  on  the  pilot  questionnaire  would  provide  information  needed  to 
design  the  final  questionnaire. 

2.  Pilot  Interview 

Six  PM  stakeholder  VV&A  process  participants  were  interviewed  using  the  pilot 
questionnaire.  Four  of  the  VV&A  participants  were  interviewed  as  a  group,  since  they 
were  located  at  the  same  U.S.  Navy  activity  and  it  was  more  convenient  to  schedule  a 
meeting  at  the  facility  (as  the  researchers  were  required  to  travel  to  conduct  the 
interviews).  The  participants  completed  the  questionnaires  before  the  interviews  and 
used  the  meeting  with  the  researchers  to  discuss  or  clarity  their  written  answers.  The 
researchers  learned  from  the  interviews  and  the  data  that  the  questionnaire/interview 
process  would  work  more  efficiently  if  the  number  of  questions  were  reduced.  It  was 
recognized  that  24  questions  would  produce  too  much  data  to  process  (for  the  intended 
number  of  stakeholders  to  be  interviewed)  using  the  pilot  questionnaire.  Also,  it  was 
realized  that  each  question  should  produce  answers  with  a  numerical  score  as  well  as  a 
narrative  comment.  The  numerical  score  would  assist  in  statistical  analysis  of  the 
stakeholder’s  answers  and  get  the  stakeholder  to  commit  to  the  assignment  of  a  grade 
(“1”  through  “5”)  to  the  answer  provided.  At  the  same  time,  the  researchers  still  desired 
narrative  comments,  since  they  would  provide  valuable  insights  as  to  why  they  assigned  a 
certain  score  to  their  answer.  Further,  the  researcher  expected  that  many  of  the  comments 
would  provide  input  for  the  conclusions  and  recommendations  expected  from  the  thesis. 
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Review  and  reflection  on  the  substance  of  the  data  collected  by  pilot  questionnaire 
influenced  the  thesis  research  questions  and  design  of  the  ten  questions  administered  by 
the  final  questionnaire.  The  researchers,  by  interviewing  just  PM  stakeholders  in  the 
pilot,  realized  the  importance  of  also  interviewing  Tester/Accreditor  and  Resource 
Sponsor  stakeholders  (in  roughly  equal  numbers  to  the  PMs  interviewed),  and  saw  the 
additional  benefit  of  interviewing  at  least  one  VV&A  Standards  Official  stakeholder.  By 
this  approach,  the  researchers  expected  to  collect  data  that  would  be  more  representative 
of  the  perspectives  held  by  VV&A  community  at  large. 


3.  Final  Questionnaire 

The  researchers  designed  a  final  questionnaire  containing  ten  questions,  each  of 
which  requested  the  stakeholder  to  answer  the  question  by  assigning  a  score  between  1 
and  5  (inclusive)  and  providing  a  comment  that  explained  the  score.  (An  exception  to  the 
1:1  score/comment  answer  pairing  was  Question  1,  which  asked  two  subsidiary  questions 
in  order  to  elicit  additional  comments  for  the  score  assigned.)  Extra  space  was  provided 
on  the  questionnaire  for  the  stakeholders  to  express  their  opinions  on  the  VV&A  process, 
regardless  of  areas  suggested  by  the  questions.  These  three  features  of  the  questionnaire: 
quantitative  scores  to  specific  questions,  narrative  comments  on  the  same  questions,  and 
freedom  to  make  additional  comments,  provided  the  researchers  with  an  instrument  for 
conducting  semi-structured  interviews  of  stakeholders.  The  ten  questions  are  provided  in 
the  Appendix. 

The  basic  purpose  of  the  ten  questions  was  to  obtain  data  from  representative, 
experienced  stakeholders  that  answer  the  primary  thesis  question  and  the  four  subsidiary 
questions  (answers  which  address  the  objectives  of  the  thesis).  Of  the  ten  questions,  the 
first  was  designed  to  capture  the  level  of  experience  with  the  COMOPTEVFOR  VV&A 
instruction  and  thus  support  the  credibility  of  the  stakeholders’  answers  to  the  remaining 
questions.  The  second  question  solicited  the  stakeholders’  basic  perceptions  of  overall 
success  in  complying  with  COMOPTEVFORINST  5000. 1,  which  went  to  the  heart  of  the 
primary  thesis  question.  The  next  three  questions  (numbers  3  through  5)  individually 
requested  the  stakeholders’  opinions  of  the  benefits,  costs,  and  risks  associated  with 
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instruction  compliance.  Conversely,  questions  6  through  8  inquired  about  the 
stakeholders’  opinions  of  the  benefits,  costs,  and  risks  associated  with  instruction  non- 
compliance.  Thus,  this  middle  set  of  six  questions  were  designed  to  cover  all  sides  of  the 
issue  and  not  presuppose  whether  all  stakeholders  saw  benefits  (vs.  liabilities)  to 
following  the  COMOPTEVFOR  prescribed  VV&A  process,  as  practiced.  Then  question 
number  9  was  asked,  in  recognition  that  COMOPTEVFOR  “recently”  updated  their 
instruction.  It  was  anticipated  that  most  of  the  stakeholders  would  report  that  the 
majority  of  their  VV&A  case  experience  had  occurred  under  the  old  instruction.  Yet  the 
researchers  also  expected  that  most  stakeholders  would  be  aware  of  what  changed  in  the 
new  instruction  and  would  be  prepared  to  comment  on  whether  the  new  instruction 
represented  an  improvement  in  the  way  VV&A  is  conducted.  Finally,  question  10  was 
designed  to  obtain  perspectives  on  opportunities  (if  any)  for  further  revision  of  the 
instruction. 

4.  Semi-Structured  Interviews 

Using  the  final  questionnaire,  the  researchers  interviewed  fourteen  stakeholders 
who  participated  in  the  VV&A  of  M&S  for  OT  of  U.S.  Navy  Surface  Ships  through  one 
of  these  four  stakeholder  roles:  (a)  Tester/Accreditor;  (b)  Resource  Sponsor:  (c)  Program 
Manager;  and  (d)  VV&A  Standards  Official.  Additionally,  each  stakeholder’s  relative 
level  of  management  responsibility  within  their  command  -  as  either  (i)  Lower,  (ii) 
Middle,  or  (iii)  Upper  management  -  was  identified.  The  role  /  level  breakout  of  the 
stakeholders  interviewed  is  shown  in  Table  A.  1-0  of  the  Appendix. 

The  researchers,  for  the  most  part,  contacted  the  stakeholders  by  e-mail  to  arrange 
the  interviews.  The  questionnaire  was  e-mailed  ahead  of  the  interviews.  Many  of  the 
stakeholders  filled  out  their  questionnaires  in  advance  of  their  interview  meeting.  The 
interviews  were  conducted  on  a  non- attribution  basis,  where  the  stakeholders’  names  and 
U.S.  Navy  commands  were  not  identified  in  order  to  protect  their  confidentiality.  The 
researchers  believed  that  there  would  be  a  better  chance  that  the  stakeholders  would 
provide  frank  opinions  if  their  names  or  activities  were  not  to  be  published.  The 
researcher  recorded  their  notes  of  the  interview  on  the  questionnaire  for  each  stakeholder. 
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Thus,  some  of  the  data  recorded  on  the  questionnaire  was  written  by  the  stakeholder  and 
some  was  recorded  by  the  researcher.  After  administering  each  question  in  order,  the 
stakeholder  was  encouraged  to  bring  up  any  other  VV&A  issue  that  came  to  mind.  These 
comments  were  recorded  in  the  space  below  question  10  (or,  if  more  room  was  needed, 
on  the  back  side  of  the  last  sheet  of  the  questionnaire).  Every  attempt  was  made  by  the 
researchers  to  get  the  stakeholders  to  provide  complete  answers  to  the  questions. 


5.  Data  Processing 

The  researchers  reviewed  the  questionnaires  and  transferred  the  numerical  scores 
and  the  raw  comments  to  an  MS  Excel  spreadsheet  to  create  a  “Results  Matrix.”  Each 
stakeholder  (using  a  letter  designator)  was  assigned  a  row.  Each  answer  was  assigned  a 
column.  The  scores  for  each  question  were  tabulated  down  the  question’s  column, 
summed,  and  averaged  for  the  question.  The  individual  raw  comment  was  loaded  into 
the  cell  adjacent  to  the  numerical  score.  This  provided  the  analyst  the  facility  of  seeing 
the  comment  with  the  score. 

a.  Identification  of  the  Stakeholder 

To  protect  their  privacy,  each  stakeholder  was  assigned  a  letter  designator 
(e.g.,  [A]  )  as  their  interview  data  was  processed  into  the  Results  Matrix.  Furthermore, 
the  stakeholder  category  (e.g.,  Tester/Accreditor  {TA}  )  and  management  level  (e.g., 
Lower  Management  {lm}  )  was  identified  using  the  letter  designators  in  brackets. 

b.  Recognition  of  the  Scores 

The  scores  recorded  on  the  questionnaires  were  transferred  to  the  Results 
Matrix.  Occasionally,  a  stakeholder  would  give  two  scores  for  a  given  question  because 
he/she  had  more  than  one  point  to  make  in  their  answer.  This  was  usually  explained  in 
the  comment  they  provided  with  the  scores.  The  researchers  reviewed  the  comment  and 
selected  the  score  that,  in  their  interpretation,  better  answered  the  true  sense  of  the 
question. 
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c.  Editing  of  the  Comments 

The  raw  data  recorded  on  the  questionnaire  was  transferred  to  the  Results 
Matrix.  After  the  matrix  was  completed  and  reviewed  for  trends,  the  comments  were 
transferred  to  an  MS  Word  document  for  editing.  By  this,  the  comments  were  corrected 
for  spelling,  grammar,  use  of  undefined  acronyms,  use  of  awkward  phrases,  attribution  to 
specific  individuals  or  commands.  Every  effort  was  taken  on  the  part  of  the  analysts  to 
preserve  the  genuine  sense  of  the  individual  comments.  The  output  of  the  comment 
editing  process  are  the  comments  provided  in  the  Appendix. 


F.  ANALYSIS  OF  DATA 

For  each  of  the  ten  interview  questions,  the  following  analysis  was  performed: 

1.  Averaging  of  Score-Answers 

The  score-answers  from  all  the  stakeholders  were  summed  and  divided  by  the 
number  of  stakeholders  to  produce  the  average  score  for  the  question.  Then,  average 
scores  were  calculated  for  each  of  the  four  stakeholder  roles  and  three  management 
levels.  These  averages  were  reported  in  a  Table.  Trends  were  detennined  from  this  data 
that  reflected  the  basic  opinions  of  the  stakeholders  by  their  stakeholder  roles  and 
management  level.  These  were  discussed  in  narrative  form. 

2.  Grouping  of  Comment-Answers 

The  comment-answers  from  the  stakeholders  were  reviewed  to  determine  whether 
a  consensus  or  divergence  of  opinions  was  apparent.  Majority  and  minority  opinions 
were  identified.  Quotes  representative  of  the  group  opinions  were  selected,  or 
paraphrased  summaries  provided.  Particularly  exceptional  comments  were  highlighted. 

3.  Sorting  of  Score- Answers 

Individual  stakeholder  score-answers  were  sorted  by  score  and  counted.  The 
score  counts  were  reported  in  a  Table.  The  count  by  score  level  was  reviewed  for  overall 
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distribution  of  scores  and  discussed  in  narrative  form.  Correlations  between  the  score 
distribution  and  comments  were  considered  and  discussed. 

G.  DEVELOPMENT  OF  CONCLUSIONS 

Conclusions  were  reached  for  each  question  and  recorded  in  Chapter  V.  These 
conclusions  were  developed  through  review  of  each  question’s  numerical  trends  and 
consideration  of  the  summarized  or  highlighted  comments.  Then,  in  Chapter  VI,  overall 
conclusions  were  developed  that  addressed  the  thesis  objectives  and  answers  were 
provided  to  the  primary  and  subsidiary  research  questions.  These  conclusions  combined 
the  results  of  the  individual  questions  and  analyzed  for  global  trends  across  the  data. 
Ultimately,  the  purpose  of  the  literature  reviews,  forum  attendances,  and  the  semi- 
structured  interviews  was  to  glean  and  combine  the  data  needed  to  address  the  thesis 
objectives  and  answer  the  research  questions.  These  outputs  were  offered  as  the  main 
conclusions  of  the  thesis. 

H.  DEVELOPMENT  OF  RECOMMENDATIONS 

Recommendations  were  developed  through  consideration  of  the  conclusions 
discussed  in  Chapter  VI.  In  addition,  the  researchers’  own  opinions,  drawing  upon  the 
lessons  learned  from  the  data,  were  indicated.  The  researchers  pointed  to  opportunities 
for  additional  research  that  became  apparent  from  the  research  but  were  not  pursued  due 
to  the  limitations  of  time  and  resources  available. 

I.  CONCLUSION 

This  chapter  described  the  eight  part  methodology  used  in  researching  the  thesis. 
The  first  step  taken  was  the  identification  of  the  major  stakeholders  who  participate  in 
this  arena  of  the  U.S.  Navy’s  VV&A  process.  Second,  information  and  data  on  the 
practices  and  expectations  of  VV&A  were  obtained  from  literature  reviews.  Attendance 
at  the  VV&A  forum  constituted  the  third  step.  Fourth,  the  researchers  developed  a 
questionnaire  and  administered  semi-structured  interviews  of  available  stakeholders. 
Fifth,  the  data  from  the  semi-structured  interviews  was  processed  into  a  Results  Matrix. 
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Sixth,  the  data  was  analyzed  for  each  of  ten  interview  questions  by  VV&A  stakeholder 
category  and  stakeholder  management  level  to  identify  trends,  etc.  By  the  seventh  step, 
conclusions  were  developed  for  both  the  individual  questions  and  for  answering  the 
primary  and  subsidiary  research  questions.  Finally,  individual  and  global 
recommendations  were  produced,  accomplishing  the  eighth  and  final  step  of  the  research 
methodology.  The  next  chapter  will  analyze  the  data  obtained  by  the  semi-structured 
interviews,  providing  the  results  of  the  thesis. 
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V.  RESULTS 


A.  INTRODUCTION 

This  chapter  reports  the  analysis  of  the  data  (reported  in  Chapter  IV)  that  was 
developed  through  the  administration  of  the  questionnaire.  Data  from  Question  1 
establishes  each  stakeholder’s  experience  with  the  COMOPTEVFOR  instruction. 
Question  2  obtains  each  stakeholder’s  assessment  of  the  level  of  compliance  observed 
with  the  instruction  and  some  supporting  evidence  for  the  assessment.  Questions  3,  4, 
and  5  solicit  the  stakeholder  regarding  the  perceived  benefits,  costs,  and  risks  associated 
with  successfully  complying  with  the  instruction.  Conversely,  Questions  6,  7,  and  8 
request  each  stakeholder’s  opinion  regarding  the  perceived  benefits,  costs,  and  risks 
associated  with  not  complying  with  the  instruction.  (Questions  2  through  8  are  meant  to 
capture  all  “sides”  of  the  compliance  issue,  depending  on  the  stakeholder’s  perspective. 
As  such,  some  of  these  questions  may  not  make  sense  to  every  stakeholder.)  Questions  9 
and  10  elicit  comments  and  recommendations  for  improvements  to  the  COMOPTEVFOR 
VV&A  instruction. 

B.  DESCRIPTION 

A  six  part  procedure  was  used  to  analyze  the  stakeholder  data: 

1 .  Purpose  of  Question:  The  purpose  of  each  question  is  reviewed. 

2.  Scores  Observed:  The  scores  are  tabulated  and  an  overall  average  score  is 
calculated  for  each  question.  Further,  the  average  score  for  each  stakeholder  role  and 
each  management  level  are  calculated  for  each  question.  These  averages  are  shown  in  a 
table  for  each  question.  Observations  are  made  about  what  these  averages  indicate  about 
the  stakeholders'  answers  to  the  question. 

3.  Stakeholder  Comments:  The  stakeholders’  comment- answers  to  each 
question  are  discussed  where:  (a)  consensus  was  obtained  or  views  diverged;  (b)  majority 
and  minority  opinions  were  evident;  and  (c)  noteworthy  ideas  were  offered. 
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4.  Score  /  Comment  Associations:  A  table  is  provided  that  shows  the 
number  of  stakeholders  that  gave  a  particular  score.  Observations  are  made  when  certain 
kinds  of  comments  correlate  with  particular  scores. 

5.  Conclusions:  Closing  observations  are  made  regarding  the  stakeholders’ 
answers  to  the  question. 

6.  Recommendations:  Suggestions  for  VV&A  process  improvements  are 
offered  within  the  context  of  the  question. 

The  above  procedure  was  performed  for  all  ten  questions.  Overarching 
observations  (trends,  etc.,  observed  across  the  answers  to  all  the  questions),  global 
conclusions,  and  final  recommendations  are  discussed  in  Chapter  VI. 

The  results  of  each  of  the  ten  interview  questions  were  analyzed,  as  described  in 
the  following  ten  sections,  C  through  L. 

C.  ANALYSIS  OF  QUESTION  #1 

Question  #1  asked:  “To  what  extent  are  you  familiar  with  the  VV&A  process 
required  by  the  newly  updated  COMOPTEVFORINST  5000.1  A  of  9  September  2004,  as 
it  applies  to  USN  surface  ship  M&S  used  for  OT&E?” 

Table  1  shows  the  selection  of  scores  that  each  stakeholder  was  able  to  choose 
from  in  answering  the  question  in  assigning  a  level  of  familiarity  with  the  instruction. 


Choices  of  Score- Answers  for  Question  #1 

Score: 

“5” 

“  4  ” 

“3” 

“2” 

66  55 

Description: 

“very 

familiar” 

“moderately 

familiar” 

“somewhat 

familiar” 

“moderately 

unfamiliar” 

“very 

unfamiliar” 

Table  1 .  Choices  of  Score- Answers  for  Question  #1 
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1.  Purpose  of  Question  #1: 

Question  #1  seeks  to  establish  each  stakeholder’s  experience  with  the 
COMOPTEVFOR  VV&A  instruction.  Among  other  insights,  the  data  obtained  is 
expected  to  help  establish  the  context  of  the  stakeholder’s  answers  to  the  remaining 
questions. 

2.  Scores  Observed  for  Question  #1: 

Table  2  shows  the  average  score-answers  to  Question  #1.  The  overall  average 
score  is  3.8,  which  indicates  that  the  typical  stakeholder  surveyed  is  “moderately 
familiar”  with  the  instruction. 


Average  Score- Answers  to  Question  #1 

Stakeholder  Role 

Management  Level  of  Stakeholder 

Avg.  Score 
within  Role 

Lower 

Middle 

Upper 

T  ester/Accreditor 

5.0 

5.0 

5.0 

5.0 

Resource  Sponsor 

2.0 

1.0 

2.0 

1.7 

Program  Manager 

4.0 

3.0 

5.0 

3.8 

VV&A  Standards  Official 

4.0 

4.0 

Avg.  Score  within  Level 

4.0 

3.0 

4.2 

3.8 

Table  2.  Stakeholder  Familiarity  with  VV&A  Instruction 


Those  performing  the  Tester/  Accreditor  role  consistently  answered  “very  familiar”  and 
scored  a  5.0  average,  the  highest  score  among  the  stakeholder  role  categories.  On  the 
other  hand,  those  performing  the  Resource  Sponsor  role  scored  an  average  of  1.7,  the 
lowest  of  the  groups,  indicating  that  they  are  typically  “moderately  unfamiliar”  with  the 
instruction.  Those  performing  the  Program  Manager  role  scored  a  3.8  average  (which  is 
the  same  as  the  overall  average),  indicating  that  they  are  typically  “moderately  familiar” 
with  the  instruction.  The  person  representing  the  VV&A  Standards  Official  role 
provided  a  score  of  4,  indicating  that  he/she  was  “moderately  familiar”  with  the 
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instruction.  The  average  scores  within  the  lower,  middle,  and  upper  management  levels 
are  4.0,  3.0,  and  4.2  respectively.  These  indicate  that  stakeholders  at  the  lower  and  upper 
management  levels  were  “moderately  familiar”  with  the  instruction,  whereas  stakeholders 
at  the  middle  management  level  are  “somewhat  familiar.” 

3.  Stakeholder  Comments  for  Question  #1: 

The  stakeholders  participated  in  a  wide  variety  of  VV&A  cases  for  OT&E  of 
USN  surface  ships,  including  efforts  that  supported  M&S  of:  Rolling  Airframe  Missile 
(RAM);  Tomahawk  Weapons  System;  Littoral  Combat  Ship  (LCS);  CVN-21;  LPD-17; 
DD(X);  ASCM  Probability  of  Raid  Annihilation  (PRA)  Test  Bed;  USS  Virginia  (SSN 
774)  Transient  Shock  Analysis  (TSA);  Joint  Lightweight  Standoff  Chemical  Agent 
Detector  (JLSCAD);  Netcentric  Warfare  Knowledge  Sharing;  LHA-6;  Threat  D  Suite; 
CEC;  ESSM;  SPYID(V);  and  ABMD.  Most  stakeholders  listed  participation  in  at  least 
three  of  these  efforts;  and  most  of  these  efforts  were  listed  by  more  than  one  of  the 
stakeholders  interviewed. 

The  stakeholders  performed  particular  roles  in  these  VV&A  cases,  including: 
Operational  Test  Director  (OTD);  participant  on  Simulation  Control  Panels  (SCPs); 
reviewer  of  VV&A  documents;  producer  of  the  ‘Guide  for  U.S.  Navy  Surface  Ship 
VV&A  of  M&S  for  OT&E’;  manager  of  COMOPTEVFOR’s  VV&A  program;  author  / 
contributor  /  reviewer  /  approver  of  COMOPTEVFORINST  5000.1  A  (issued  9 
September  2004);  developer  of  long-term  strategic  policy  with  respect  to  the  U.S.  Navy 
and  OT&E;  resource  sponsor  (funding)  and  requirements  definer  (via  Capabilities  Design 
Documents)  for  U.S.  Navy  surface  ships  and/or  their  weapons  systems;  arbitrator 
between  Program  Manager  and  Tester/Accreditor  with  regard  to  balancing  M&S  costs 
and  risks;  oversight  of  surface  ship  Test  &  Evaluation  Master  Plan  (TEMP)  process; 
oversight  of  the  U.S.  Navy’s  T&E  efforts  (including  M&S  for  T&E);  project  engineer  / 
program  analyst  for  surface  ship  acquisition  program  office  developing  M&S;  developer 
of  M&S  and  contributor  to  VV&A  documentation;  manager  of  the  VV&A  process  for 
U.S.  Navy  surface  ship  /  weapons  system  acquisition  Program  Manager;  oversight  of 
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PM’s  VV&A  process;  oversight  of  the  U.S.  Navy’s  VV&A  processes;  and  communicator 
/  coordinator  of  the  U.S.  Navy’s  VV&A  standards. 

The  older  (superseded)  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  in  effect  for  most  the  cases  contributed  by  the  stakeholders.  However,  the  newer 
instruction  5000.1  A  was  in  effect  for  the  cases  started  since  9  September  2004. 

4.  Score  /  Comment  Associations  for  Question  #1: 

No  particular  correlation  was  seen  between  the  stakeholder  scores  and  the  types  of 
experiences  (that  is,  the  kind  of  ship  or  weapon  system  VV&A’ed)  cited  by  the 
stakeholders.  Table  3  shows  the  distribution  of  score-answers  for  Question  #1. 
However,  there  were  strong  associations  between  scores  of  5  (of  being  “very  familiar” 
with  the  instruction)  and  the  particular  roles  of  (a)  Tester/Accreditor  reviewer  of  VV&A 
documents;  (b)  manager  of  a  Command’s  VV&A  program;  and  (c)  contributor  /  reviewer 
of  COMOPTEVFORINST  5000.1  A.  There  were  also  strong  associations  between  the 
lowest  scores  of  2  and  1  (of  either  being  “moderately  unfamiliar”  or  “very  unfamiliar” 
with  the  instruction)  and  the  particular  roles  of  (a)  providing  program  funding  /  defining 
acquisition  requirements  and  (b)  arbitrating  between  the  program  managers  and  the 
Tester/Accreditors  with  regard  to  balancing  M&S  costs  and  risks. 


Distribution  of  Question  #1  Score -Answers 

Level  of  Familiarity  with  VV&A  Instruction 

Count 

Stakeholders  who  provided  a  score  of  “5” 

6 

Stakeholders  who  provided  a  score  of  “4” 

4 

Stakeholders  who  provided  a  score  of  “3” 

0 

Stakeholders  who  provided  a  score  of  “2” 

3 

Stakeholders  who  provided  a  score  of  “1” 

1 

Total 

14 

Table  3.  Distribution  of  Question  #1  Scores-Answers 
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5.  Conclusions  for  Question  #1: 

The  typical  stakeholder  surveyed  is  “moderately  familiar”  with  the  instruction. 
This  is  to  be  expected,  given  that  stakeholders  interviewed  had  key  roles  in  the  VV&A  of 
M&S  used  for  OT&E.  It  would  be  surprising  if  the  participants  in  this  application  of 
VV&A  were  not  familiar  with  the  instruction  since  they  would  be  expected  to  refer  to  it 
routinely  in  the  course  of  performing  their  respective  roles  in  the  VV&A  process.  The 
contrary  turns  out  to  be  the  case  only  for  Resource  Sponsors,  who  typically  indicated  that 
they  are  “moderately  unfamiliar”  with  the  instruction.  We  learned  from  the  interviews 
that  the  typical  Resource  Sponsor  does  not  believe  that  “intimate”  familiarity  with  the 
COMOPTEVFOR  VV&A  instruction  is  necessary  to  their  role.  Rather,  they  believe  their 
function  is  to  be  generally  familiar  with  the  VV&A  requirement  so  as  to  help  balance  the 
high  technical  expectations  for  the  M&S,  held  by  the  Tester/Accreditors,  with  the  desire 
to  control  the  cost  and  schedule  to  produce  the  M&S,  held  by  the  Program  Managers. 

D.  ANALYSIS  OF  QUESTION  #2: 

Question  #2  asked:  “What  level  of  compliance  with  the  (effective)  instruction  do 
you  think  was  achieved  in  this  case?” 

Table  4  shows  the  selection  of  scores  that  each  stakeholder  was  able  to  choose 
from  in  answering  the  question,  assigning  a  level  of  compliance  with  the  instruction. 


Choices  of  Score- Answers  for  Question  #2 

Score: 

“5” 

“  4  ” 

“3” 

“2” 

66  55 

Description: 

“100-91% 

compliance” 

“90-81% 

compliance” 

“80-71% 

compliance” 

“70-61% 

compliance” 

“60%  or  less 
compliance” 

Table  4.  Choices  of  Score-Answers  for  Question  #2 


66 


1.  Purpose  of  Question  #2: 

Question  #2  requests  the  stakeholder’s  assessment  of  the  level  of  compliance 
observed  for  the  COMOPTEVFOR  instruction  and  expects  some  supporting  evidence  for 
the  assessment.  The  answer  to  this  question  provides  the  basis  for  comparing  the 
practices  of  VV&A  under  the  COMOPTEVFORINST  standard  and  judging  success. 

2.  Scores  Observed  for  Question  #2: 

Table  5  shows  the  average  score-answers  to  Question  #2.  The  overall  average 
score  is  3.9,  which  indicates  that  the  typical  stakeholder  surveyed  sees  “90-81% 
compliance”  with  the  instruction.  Those  performing  the  Tester/Accreditor  role  generally 
answered  “80-71%  compliance”  and  scored  a  3.2  average,  the  lowest  score  among  the 
stakeholder  role  categories.  On  the  other  hand,  the  stakeholder  representing  the  VV&A 
Standards  Official  role  provided  a  score  of  5,  the  highest  score  among  the  groups, 
indicating  that  he/she  thinks  that  “100-91%  compliance”  was  achieved  for  the  instruction. 
Those  performing  the  Program  Manager  role  scored  a  4.4  average  (which  is  0.5  above  the 
overall  average),  indicating  that  they  typically  see  at  least  “90-81%  compliance”  with  the 
instruction.  The  Resource  Sponsors  scored  an  average  of  3.7,  which  is  just  0.2  below  the 
overall  average.  The  average  scores  within  the  lower,  middle,  and  upper  management 
levels  are  4.6,  3.5,  and  3.4  respectively.  These  scores  indicate  that  stakeholders  at  the 
lower  management  level  see  a  noticeably  higher  level  of  compliance  (at  “100-91% 
compliance”)  with  the  instruction,  whereas  stakeholders  at  the  middle  and  upper 
management  levels  see  only  “90-81%  compliance.” 

3.  Stakeholder  Comments  for  Question  #2: 

The  stakeholders’  comment-answers  indicated  that  no  consensus  exists  regarding 
the  degree  of  compliance  with  the  VV&A  instruction.  Instead,  the  comments  suggest 
that  there  are  two  camps  of  experience.  The  first  camp  saw  a  fairly  high  level  of 
compliance  and  pointed  to  a  number  of  examples.  One  striking  example  was  recounted 
by  a  Resource  Sponsor,  as  follows: 
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Average  Score- Answers  to  Question  #2 

Stakeholder  Role 

Management  Level  of  Stakeholder 

Avg.  Score 
within  Role 

Lower 

Middle 

Upper 

T  ester/Accreditor 

5.0 

4.0 

1.0 

3.2 

Resource  Sponsor 

5.0 

1.0 

5.0 

3.7 

Program  Manager 

4.0 

4.5 

5.0 

4.4 

VV&A  Standards  Official 

5.0 

5.0 

Avg.  Score  within  Level 

4.6 

3.5 

3.4 

3.9 

Table  5.  Observed  Level  of  Compliance  with  VV&A  Instruction 


A  recently  completed  VV&A  effort  enabled  the  elimination  of  a 
$70M  physical  T&E  event  required  for  a  major  ship  acquisition. 
This  was  achieved  because  the  PM  employed  a  “test-model-test” 
iterative  effort  over  a  number  of  years  in  the  ship’s  RDT&E 
program  to  collectively  understand  the  physics  of  the  event  on  the 
ship.  Further,  the  ship’s  designs,  tests,  and  models  were  planned  to 
work  in  support  of  each  other.  The  ship’s  subassembly 
construction  techniques  facilitated  M&S  and  VV&A.  Finally, 
M&S  process  control  was  achieved  to  at  least  CMMI  level  3 
standards.  This  facilitated  a  rapid  VV&A  process. 


Included  in  this  first  camp  was  an  acknowledgement  by  many  that,  though  overall 

compliance  was  high,  there  was  less  success  at  delivering  the  V&V  products  on  time. 

(The  stakeholders  went  on  to  say  that  the  lateness  of  documentation  was  usually  not 

critical  to  the  process.)  On  the  other  hand,  the  second  (though  smaller)  camp  perceived  a 

low  level  of  compliance  by  the  M&S  proponent,  citing  cases  such  as  this: 

...  the  Resource  Sponsor  ran  into  a  big  issue  at  the  end  of  one 
VV&A  process:  DOT&E  was  not  satisfied  with  the  capability  of 
the  M&S  to  credibly  predict  the  performance  of  the  weapon  system 
modeled.  The  Resource  Sponsor  wanted  to  use  M&S  to  avoid  the 
cost  and  risk  entailed  in  conducting  a  certain  close-in  live-fire 
testing  event.  But  the  PM  had  not  engaged  the  Tester/Accreditor 
sufficiently  (or  early  enough)  in  their  VV&A  process  to  identify 
the  shortcomings  of  the  M&S,  such  that  DOT&E  rejected  the  use 
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of  the  M&S  for  the  close-in  event.  Consequently,  the  Resource 
Sponsor  had  to  reprogram  $250M  to  conduct  a  physical  test  that 
the  M&S  was  hoping  to  avoid. 


This  second  camp  also  observed  that  “many  PMs  did  not  seem  to  be  knowledgeable 
about  the  instruction.” 

Beyond  the  individual  cases  mentioned,  both  camps  offered  reasons  why 

proponents,  when  highly  compliant,  were  successful.  Reasons  for  success  included:  “The 

PMs  used  COMOPTEVFOR’s  Surface  Warfare  VV&A  Guide;”  and  “The  PMs  used  the 

process  diagram  developed  by  NDAT.”  One  particularly  noteworthy  comment  on  the 

path  to  successful  compliance  was  offered,  as  follows: 

The  maturity  and  experience  of  the  team  that  develops  the  ship’s 
T&E  program,  which  includes  M&S  in  its  mix,  has  a  lot  to  do  with 
any  successful  compliance  with  the  VV&A  instruction. 

Finally,  it  was  observed  that  the  PMs  generally  had  good  intentions  to  comply 
with  the  instruction,  as  demonstrated  by  their  efforts  to  learn  the  process  through 
attendance  at  VV&A  forums  and  working  with  NMSO  to  understand  the  mechanics  of 
the  VV&A  process,  including  standard  documentation  requirements. 

4.  Score  /  Comment  Associations  for  Question  #2: 

Table  6  shows  that  stakeholders’  scores  fall  into  two  distinct  groups.  The 
majority  (1 1  of  14)  of  stakeholders  gave  scores  of  5  and  4,  indicating  that  a  high  level  of 
instruction  compliance  was  experienced.  On  the  other  hand,  a  vocal  minority  (3  of  14) 
scored  compliance  at  “1”  and  commented  that  some  M&S  proponents  were  deficient  in  at 
least  one  critical  area  of  the  VV&A  process.  “In  one  case,  the  PM  did  not  provide  the 
data  to  finish  the  VV&A  package.  So  the  data  had  to  be  developed  by  the  OT  himself.” 
The  lack  of  “3”  and  “2”  scores  indicate  that  there  appears  to  be  no  middle  ground  among 
the  stakeholders  regarding  level  of  compliance  observed. 
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Distribution  of  Question  #2  Score -Answers 

Level  of  Compliance  with  VV&A  Instruction 

Count 

Stakeholders  who  provided  a  score  of  “5” 

7 

Stakeholders  who  provided  a  score  of  “4” 

4 

Stakeholders  who  provided  a  score  of  “3” 

0 

Stakeholders  who  provided  a  score  of  “2” 

0 

Stakeholders  who  provided  a  score  of  “1” 

3 

Total 

14 

Table  6.  Distribution  of  Question  #2  Scores-Answers 


5.  Conclusions  for  Question  #2: 

With  an  overall  stakeholder  average  score  of  3.9  (out  of  a  perfect  5.0)  and  noting 
that  1 1  out  of  14  stakeholders  gave  scores  of  at  least  4,  it  appears  that  most  stakeholders 
think  that  there  was  a  moderately  high  level  of  compliance  with  COMOPTEVFORINST 
5000.1  A.  On  the  other  hand,  a  vocal  minority  felt  strongly  that  there  were  clear  instances 
of  non-compliance  and  provided  evidence  and  rationale  for  these  observations.  Some 
non-compliance  issues  were  also  noted  within  the  overall  positive  comments  of  the 
majority  (e.g.,  the  high  level  of  compliance  achieved  with  respect  to  “content,”  though 
not  as  high  with  respect  to  “schedule”).  Some  stakeholders  expressed  complex  scores, 
which  indicated  positives  for  some  areas  of  VV&A  performance  and  admitted  negatives 
for  others.  In  general,  it  appears  that  stakeholders  frankly  expressed  their  opinions  and 
freely  pointed  to  discrepancies.  Thereby  is  indicated  that  there  is  room  for  improvement 
in  the  duty  of  complying  with  the  COMOPTEVFOR  VV&A  instruction. 

E.  ANALYSIS  OF  QUESTION  #3 

Question  #3  asked:  “What  do  you  see  to  be  the  benefits  associated  with 
compliance  with  this  instruction?” 
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Table  7  shows  the  selection  of  scores  that  each  stakeholder  was  able  to  choose 
from  in  answering  the  question,  assigning  a  level  of  benefits  seen  in  complying  with  the 
instruction. 


Choices  of  Score- Answers  for  Question  #3 

Score: 

“5” 

“  4  ” 

“3” 

“2” 

66  59 

Description: 

“many 

significant 

benefits” 

“several 

significant 

benefits” 

“one 

significant 

benefit” 

“a  few 
minor 
benefits” 

“no 

real 

benefit” 

Table  7.  Choices  of  Score-Answers  for  Question  #3 


1.  Purpose  of  Question  #3: 

Question  #3  solicits  the  stakeholder’s  perception  of  the  benefits  associated  with 
successfully  complying  with  the  COMOPTEVFOR  instruction  and  expects  some 
supporting  examples  for  the  assessment.  The  answer  to  this  question  is  likely  to  provide 
the  most  compelling  reasons  for  supporting  the  VV&A  process  advocated  by 
COMOPTEVFORINST  5000.1  A. 

2.  Scores  Observed  for  Question  #3: 

Table  8  shows  the  average  score-answers  to  Question  #3.  The  overall  average 
score  is  4.5,  which  indicates  that  the  typical  stakeholder  surveyed  sees  “many  significant” 
benefits  associated  with  compliance  with  this  instruction.  This  strong  positive  on 
benefits  is  particularly  seen  for  those  who  perform  the  Tester/Accreditor  role  and  those  at 
the  Middle  and  Upper  Management  levels,  who  scored  averages  of  4.8,  4.8,  and  4.6 
respectively  (the  highest  scores  among  the  stakeholder  role  categories).  The  lowest 
group  score  is  4.0,  which  comes  from  the  VV&A  Standards  Official  role,  who  sees 
“several  significant”  benefits  in  complying  with  the  instruction.  Program  Managers, 
Resource  Sponsors,  and  those  at  Lower  Management  levels  scored  averages  of  4.4,  4.3, 
and  4.2  respectively.  In  sum,  the  scores  were  tightly  grouped  between  4.0  and  5.0, 
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indicating  that  all  stakeholders  perceive  significant  benefits,  which  range  from  “several” 


to  “many.” 


Average  Score- Answers  to  Question  #3 

Stakeholder  Role 

Management  Level  of  Stakeholder 

Avg.  Score 
within  Role 

Lower 

Middle 

Upper 

T  ester/Accreditor 

5.0 

5.0 

4.5 

4.8 

Resource  Sponsor 

4.0 

4.0 

5.0 

4.3 

Program  Manager 

3.5 

5.0 

5.0 

4.4 

VV&A  Standards  Official 

4.0 

4.0 

Avg.  Score  within  Level 

4.2 

4.8 

4.6 

4.5 

Table  8.  Stakeholder  Perceived  Benefits  of  Compliance  with  VV&A  Instruction 

3.  Stakeholder  Comments  for  Question  #3: 

The  stakeholders’  comment- answers  indicated  that  a  strong  consensus  exists 
regarding  the  high  number  and  kind  benefits  of  complying  with  the  VV&A  instruction. 
Every  stakeholder  provided  examples  that  highlighted  at  least  one  significant  benefit,  and 
nine  of  14  stakeholders  mentioned  two  or  more  benefits.  Eight  benefits  of  compliance 
with  COMOPTEVFORINST  5000.1  A  cited  were: 

a.  Better  assurance  that  M&S  will  meet  the  needs  of  the  user. 

b.  Higher  quality  products  are  delivered  in  a  shorter  time. 

c.  Financial  savings  due  to  management  efficiencies. 

d.  Accreditation  follows  naturally  from  the  proponent’s  V&V  effort. 

e.  Avoids  program  delays  in  reaching  Acquisition  Milestone  B. 

f.  Avoids  the  need  to  reprogram  RDT&E  funds  to  cover  more  OT. 

g.  Promotes  standardization,  commonality,  and  M&S  reuse. 

h.  M&S  and  OT  problems  are  identified  and  defined  early. 
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As  one  stakeholder  summed  up,  “The  bottom-line  is  that  complying  with  the  instruction 
successfully  obtains  the  Accreditor’s  acceptance  of  the  use  of  the  M&S.” 


4.  Score  /  Comment  Associations  for  Question  #3: 

Table  9  shows  that  the  stakeholders  scored  the  benefits  of  compliance  highly, 

where  the  single  lowest  score  of  “3”  indicates  at  least  “one  significant  benefit.”  A  super- 

majority  (13  of  14)  of  the  stakeholders  saw  “several  to  many  significant  benefits.”  in 

compliance  with  instruction.  One  comment  described  a  nexus  of  benefits  realized  from 

long-term  program  investment  in  M&S  and  adherence  to  COMOPTEVFOR’s  process: 

One  particular  U.S.  Navy  weapons  acquisition  program  made 
heavy  use  of  M&S  for  tens  of  years.  What  is  in  place  today  has 
saved  a  lot  of  money.  The  program  has  end-to-end  testing  and 
M&S  capability,  which  provides  flexibility  in  using  different 
modes,  and  uses  an  iterative  model  update  process.  The  program 
is  able  to  work  a  combination  of  modeling  and  simulation,  where 
the  simulation  generates  environmental  inputs  into  real  hardware 
so  that  the  weapon’s  mission  can  be  run  in  a  very  realistic  way. 


Distribution  of  Question  #3  Score -Answers 

Level  of  Benefits  Perceived  with  Compliance 

Count 

Stakeholders  who  provided  a  score  of  “5” 

8 

Stakeholders  who  provided  a  score  of  “4” 

5 

Stakeholders  who  provided  a  score  of  “3” 

1 

Stakeholders  who  provided  a  score  of  “2” 

0 

Stakeholders  who  provided  a  score  of  “1” 

0 

Total 

14 

Table  9.  Distribution  of  Question  #3  Scores-Answers 


5.  Conclusions  for  Question  #3: 

With  an  overall  average  score  of  4.5  (out  of  5.0)  and  a  multitude  of  benefits  cited, 
it  is  clear  that  a  strong  consensus  exists  among  the  VV&A  stakeholder  community  that 
compliance  with  the  COMOPTEYFOR  instruction  yields  many  significant  benefits.  The 
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most  obvious  benefit  of  compliance  cited  was  the  high  likelihood  of  Tester/Accreditor 
acceptance  of  the  proponent’s  M&S  for  the  use  in  addressing  a  COI  under  OT.  The 
comments  revealed  that  many  other  program  benefits  result  from  following  the  process 
that  achieves  this  bottom-line  benefit.  In  large  part,  the  stakeholders  recognized  that 
early  interaction  between  the  PM  and  COMOPTEVFOR  was  key  to  meeting  OT 
requirements  and  avoiding  any  delays  to  reaching  program  milestones.  Many  saw  overall 
program  cost  reduction  and  standardization  of  M&S  products  as  significant  benefits 
realized  through  compliance.  That  all  stakeholders  interviewed  were  readily  able  to 
identify  some  benefits  of  compliance  provides  reason  to  believe  that  COMOPTEVFOR’s 
instruction  is  basically  successful. 


F.  ANALYSIS  OF  QUESTION  #4 

Question  #4  asked:  “What  do  you  see  to  be  the  costs  associated  with  compliance 
with  this  instruction?” 

Table  10  shows  the  selection  of  scores  that  each  stakeholder  was  able  to  choose 
from  in  answering  the  question,  assigning  a  level  of  costs  seen  in  complying  with  the 
instruction. 


Choices  of  Score- Answers  for  Question  #4 

Score: 

“5” 

“  4  ” 

“3” 

“2” 

(.(.  59 

“many 

“several 

“one 

“a  few 

“no 

Description: 

significant 

significant 

significant 

minor 

real 

costs” 

costs” 

cost” 

costs” 

cost” 

Table  10.  Choices  of  Score- Answers  for  Question  #4 


1.  Purpose  of  Question  #4: 

Question  #4  solicits  the  stakeholder’s  perception  of  the  costs  associated  with 
obediently  complying  with  the  COMOPTEVFOR  instruction  and  expects  some 
supporting  examples  for  the  assessment.  The  answer  to  this  question  is  likely  to  cite  the 
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negative  financial  impacts  of  dutiful  compliance  with  an  instruction  that  leaves  the  issue 
open  for  the  M&S  proponent  to  address  and  the  sponsor  to  support. 

2.  Scores  Observed  for  Question  #4: 

Table  1 1  shows  the  average  score-answers  to  Question  #4.  The  overall  average 
score  is  2.4,  which  indicates  that  the  typical  stakeholder  sees  “a  few  minor  costs”  in 
complying  with  the  instruction.  But  a  further  breakdown  of  the  scores  reveals  a  strong 
divergence  of  opinion  between  subcategories  of  stakeholder  role  and  management  level. 
Regarding  those  performing  the  Tester/Accreditor  role,  it  would  seem  from  their  group 
average  score  of  2.6  that  their  general  outlook  on  the  cost  of  compliance  falls  somewhere 
between  “a  few  minor  costs”  and  “one  significant  cost.”  However,  this  generalization 
would  mask  a  clear  difference  of  opinion  within  the  management  levels  of  that 
stakeholder  role.  The  Tester/Accreditors  are  split  between  their  members  at  the  lower 
management  level,  who  scored  an  average  of  4.0  (thereby  indicating  “several  significant 
costs”)  and  the  organization’s  middle  and  upper  level  managers,  who  scored  averages  of 
1.0  and  2.0  respectively  (thereby  indicating  “no  real  cost”  or  “a  few  minor  costs” 
respectively).  Similarly,  the  Program  Manager  stakeholder  group,  which  averaged  a  2.8 
score,  is  divided  into  two  factions.  In  this  instance,  the  PM’s  lower  management  level 
does  not  perceive  more  than  “a  few  minor  costs,”  providing  a  group  average  score  of  1.5. 
On  the  other  hand,  the  middle  and  upper  management  levels  of  the  PM  group  “several 
significant  costs”  with  average  scores  of  3.5  and  4.0,  respectively.  In  contrast,  there  is 
fair  unanimity  of  opinion,  “few  to  no  costs”  among  the  Resource  Sponsors  (1.7  average 
score)  and  also  the  VV&A  Standards  Official  group  (scoring  1.0),  regardless  of 
management  level. 

3.  Stakeholder  Comments  for  Question  #4: 

There  was  a  divergence  of  opinions  expressed  among  the  stakeholders’  comment- 
answers.  However,  this  disagreement  can  be  grouped  into  two  like-minded  camps.  The 
majority  (9  of  14)  of  stakeholders  felt  that  only  a  few  minor  costs  (if  any)  were  truly 
added  to  U.S.  Navy  ship  acquisition  programs’  costs  of  delivering  M&S  products  useful 
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for  OT.  Two  comments  were  typical  of  the  majority  opinion.  The  first  point  was  made 

by  a  member  of  the  Tester/Accreditor  group,  as  follows: 

Ultimately,  there  are  no  genuine  added  costs  associated  with 
compliance.  It  could  be  argued,  further,  that  the  total  cost  to  use 
M&S  for  OT&E  is  probably  less  than  that  which  would  be  needed 
to  perfonn  physical  testing. 


Average  Score- Answers  to  Question  #4 

Stakeholder  Role 

Management  Level  of  Stakeholder 

Avg.  Score 
within  Role 

Lower 

Middle 

Upper 

T  ester/Accreditor 

4.0 

1.0 

2.0 

2.6 

Resource  Sponsor 

2.0 

2.0 

1.0 

1.7 

Program  Manager 

1.5 

3.5 

4.0 

2.8 

VV&A  Standards  Official 

1.0 

1.0 

Avg.  Score  within  Level 

2.6 

2.5 

2.0 

2.4 

Table  1 1 .  Stakeholder  Perceived  Costs  of  Compliance  with  VV&A  Instruction 


The  second  such  comment  was  offered  by  a  Resource  Sponsor,  to  wit: 

Compliance  with  the  instruction  means  more  “up-front  time”  and 
man-hours  devoted  to  PM  -  COTF  interactions  than  would  the 
typical  PM  (in  all  likelihood)  devote  to  VV&A  conducted  without 
the  obligation  of  the  instruction.  However,  without  the  instruction, 
the  PM  would  end  up  having  to  collect  more  ‘real’  data  (thereby 
incurring  more  cost)  to  make  the  M&S  credible. 


Comments  provided  by  minority,  on  the  other  hand,  pointed  out  the  most  palpable  of  the 

costs  incurred  by  the  PM,  such  as  this  concern: 

The  stringent  documentation  and  configuration  management 
requirements  demanded  by  the  instruction  will  most  likely  increase 
costs  for  the  developers. 

That  said,  it  seems  that  minority  still  recognized  the  potential  of  COMOPTEVFOR’s 
VV&A  process  to  avoid  help  avoid  added  RDT&E  costs  and  actually  save  overall 
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acquisition  costs.  A  good  “take-away”  quote  that  expressed  this  sentiment  was  offered 
by  the  VV&A  Standards  Official  stakeholder:  “Compliance  with  the  instruction  really 
provides  a  savings  to  the  U.S.  Navy.” 


4.  Score  /  Comment  Associations  for  Question  #4: 

Review  of  Table  12  distribution  of  score-answers  confirms  that  there  appears  to 
be  two  groups  of  opinion  regarding  the  cost  of  compliance.  The  minority  (5  of  14)  group, 
seeing  one  to  several  significant  costs  (scores  of  3  and  4),  expressed  concern  about 
having  to  pay  more  to  produce  “nice-to-have”  high-fidelity  M&S  and  detailed  VV&A 
documentation.  On  the  other  hand,  the  majority  (9  of  14)  saw  only  a  few  (if  any)  minor 
costs  associated  with  compliance,  but  also  admitted  that  performing  VV&A  incurs  more 
“upfront  costs.”  As  one  stakeholder  offered,  “When  it  comes  to  compliance,  there  is  no 
free  lunch!” 


Distribution  of  Question  #4  Score -Answers 

Level  of  Costs  Perceived  with  Compliance 

Count 

Stakeholders  who  provided  a  score  of  “5” 

0 

Stakeholders  who  provided  a  score  of  “4” 

4 

Stakeholders  who  provided  a  score  of  “3” 

1 

Stakeholders  who  provided  a  score  of  “2” 

5 

Stakeholders  who  provided  a  score  of  “1” 

4 

Total 

14 

Table  12.  Distribution  of  Question  #4  Scores-Answers 


5.  Conclusions  for  Question  #4: 

The  difference  of  opinion  among  stakeholders  as  to  whether  significant  costs  are 
incurred  by  the  PM  in  diligently  complying  with  COMOPTEVFORINST  5000.1  A 
indicates  that  some  further  changes  may  be  needed  in  the  way  VV&A  process  is  managed 
and  its  costs  budgeted.  Not  all  the  stakeholders,  who  presumably  work  together  at  the 
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key  points  in  the  process,  see  eye  to  eye  on  the  financial  investments  demanded  by  the 
process.  It  is  also  interesting  to  note  that  stakeholders  from  the  lower  management  level 
of  the  Tester/Accreditor  group,  who  would  normally  not  be  expected  to  be  sensitive  to 
costs,  identified  “several  significant  costs”  in  complying  with  the  COMOPTEVFOR 
instruction.  It  may  be  that,  by  their  closer  working  relationships  with  their  M&S 
proponent  counterparts,  they  “feel  the  pain”  of  the  costs  carried  by  the  Program  Managers 
as  they  undergo  the  VV&A  process.  Of  course,  the  PMs  at  the  middle  and  upper  levels 
of  management,  who  scored  averages  of  3.5  and  4.0,  would  be  very  much  expected  to  be 
sensitive  to  the  costs  of  compliance,  since  they  must  balance  their  programs’  budgets. 


G.  ANALYSIS  OF  QUESTION  #5 

Question  #5  asked:  “What  do  you  see  to  be  the  risks  associated  with  compliance 
with  this  instruction?” 

Table  13  shows  the  selection  of  scores  that  each  stakeholder  was  able  to  choose 
from  in  answering  the  question,  assigning  a  level  of  risks  seen  in  complying  with  the 
instruction. 


Choices  of  Score- Answers  for  Question  #5 

Score: 

“  5  ” 

“  4  ” 

“3” 

“2  ” 

U  ^  59 

“many 

“several 

“one 

“a  few 

“no 

Description: 

significant 

significant 

significant 

minor 

real 

risks” 

risks” 

risk” 

risks” 

risk” 

Table  13.  Choices  of  Score- Answers  for  Question  #5 


1.  Purpose  of  Question  #5: 

Question  #5  solicits  the  stakeholder’s  perception  of  the  risks  associated  with 
appropriately  complying  with  the  COMOPTEVFOR  instruction  and  expects  some 
supporting  examples  for  the  assessment.  The  answer  to  this  question  is  likely  to  offer  up 
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concerns  that  stakeholders  have  while  following  the  instruction  -  uncertainties  in  the 
process  that  would  undermine  final  accreditation  of  the  proposed  M&S. 


2.  Scores  Observed  for  Question  #5: 

Table  14  shows  the  average  score-answers  to  Question  #5.  The  overall  average 
score  is  1.7,  which  indicates  that  the  typical  stakeholder  sees  “a  few  minor  risks”  in 
compliance  with  the  instruction.  The  average  scores  of  the  individual  stakeholder  roles 
held  very  close  to  the  overall  1.7  average,  with  scores  of  1.8,  1.7,  1.8,  and  1.0  for  the 
Tester/Accreditor,  Resource  Sponsor,  PM,  and  VV&A  Standard  roles,  respectively. 
Lower  and  middle  management  levels  scored  averages  of  2.0  for  every  participating 
stakeholder  role,  except  for  the  1.5  average  score  observed  for  PM  middle  management. 
Upper  management  issued  the  lowest  overall  average  score  of  1.4,  with  individual 
average  scores  of  1.5,  1.0,  2.0,  and  1.0  for  the  Tester/Accreditor,  Resource  Sponsor, 
Program  Manager,  and  VV&A  Standard  roles,  respectively,  indicating  that  “no  real  risk” 
is  taken  by  proponents  in  complying  with  the  COMOPTEVFORINST  5000. 1  A. 


Average  Score- Answers  to  Question  #5 

Stakeholder  Role 

Management  Level  of  Stakeholder 

Avg.  Score 
within  Role 

Lower 

Middle 

Upper 

T  ester/Accreditor 

2.0 

2.0 

1.5 

1.8 

Resource  Sponsor 

2.0 

2.0 

1.0 

1.7 

Program  Manager 

2.0 

1.5 

2.0 

1.8 

VV&A  Standards  Official 

1.0 

1.0 

Avg.  Score  within  Level 

2.0 

1.8 

1.4 

1.7 

Table  14.  Stakeholder  Perceived  Risks  of  Compliance  with  VV&A  Instruction 

3.  Stakeholder  Comments  for  Question  #5: 

The  stakeholder  comment-answers  indicated  that  consensus  was  obtained  on  the 
matter  of  risk  with  regard  to  complying  with  the  COMOPTEVFOR  instruction.  The  vast 
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majority  stakeholder  comments  observed  that  there  were  “few,  if  any,”  real  risks  involved 

in  the  business  of  complying  with  the  instruction.  At  the  same  time,  most  of  their 

comments  acknowledged  that  less  substantial  risks  were  present  -  just  not  as  bad  as  first 

feared  by  the  M&S  proponent.  This  comment  was  typical: 

There  are  political  risks,  which  are  inherent  in  the  business  of  VV 
&A.  Compliance  with  the  instruction  makes  some  of  the  big  risks 
go  away,  but  at  the  same  time  introduces  a  few  new,  smaller  ones. 


The  notion  of  “inherent  risk”  echoed  though  the  stakeholders’  comments,  as  shown  by 
this  statement: 


The  greatest  risk  the  PM  could  encounter  by  dutifully  complying 
with  the  instruction  would  be  to  discover,  through  the  M&S  for 
OT&E  process,  that  the  system  to  be  acquired  does  not  deliver  the 
required  capability  and  consequently  should  not  be  fielded.  But 
this  is  precisely  what  OT&E  is  intended  to  discover  and  thus  an 
inherent  risk  (whether  M&S  or  physical  testing  is  used)  for  the  PM 
who  presents  his  system  for  OT&E. 


There  was  also  this  observation  on  the  challenge  of  tackling  difficult  technical  issues: 

VV&A  planning  should  go  hand  in  hand  with  engineering  software 
development.  No  significant  risk  should  arise  with  compliance. 
However,  there  are  minor  risks  that  come  from  the  instruction’s 
effect  of  driving  PMs  to  comply  to  processes  that  produce 
relatively  ‘safe’  (less  pioneering)  outcomes  rather  than 
encouraging  these  PMs  to  tackle  difficult  technical  issues  that, 
while  they  may  not  lead  to  as  high  a  percentage  of  compliance,  still 
produce  greater  M&S  capability. 


Dissenting  opinions  were  offered  by  two  stakeholders,  who  voiced  the  existence  of  at 

least  “one  significant  risk,”  one  of  which  was  described  as  follows: 

There  is  the  risk  of  last  minute  changes  to  test  requirements  or 
objectives,  which  may  cause  delays  in  accreditation  (and  thus  test 
schedule)  because  stepping  back  the  process  will  require  rework  on 
many  artifacts  and  add  extra  V&V  activities. 


The  overall  effect  of  the  stakeholders’  comments  was  to  acknowledge  the  existences  of 
risks  but  point  out  that  they  are  very  manageable  within  the  context  of  conducting  VV&A 
in  accordance  with  COMOPTEVFOR-INST  5000.1  A. 
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4.  Score  /  Comment  Associations  for  Question  #5: 

The  distribution  of  individual  stakeholder  score-answers  shown  in  Table  15  shows 
that  all  of  the  scores  lay  at  the  bottom  of  the  assessment  of  risk  scale.  12  of  14 
stakeholders  said  that  there  were  few,  if  any,  risks  associated  with  VV&A  instruction 
compliance.  The  remaining  two  cite  “one  significant  risk”  but,  as  discussed  above,  the 
risks  are  still  viewed  to  be  manageable. 

5.  Conclusions  for  Question  #5: 

The  vast  majority  of  stakeholders  sampled  viewed  the  risk  of  complying  with  the 
instruction  as  “small”  and  “well  within  the  ability  to  manage.”  Nevertheless,  most 
stakeholders  also  acknowledged  that  “risk”  still  weighed  on  the  mind  of  the  M&S 
proponent.  The  comments  sought  to  explain  the  context  for  managing  these  risks, 
describing  them  as  “inherent”  in  the  nonnal  course  of  presenting  M&S  for  accreditation. 


Distribution  of  Question  #5  Score -Answers 

Level  of  Risks  Perceived  with  Compliance 

Count 

Stakeholders  who  provided  a  score  of  “5” 

0 

Stakeholders  who  provided  a  score  of  “4” 

0 

Stakeholders  who  provided  a  score  of  “3” 

2 

Stakeholders  who  provided  a  score  of  “2” 

6 

Stakeholders  who  provided  a  score  of  “1” 

6 

Total 

14 

Table  15.  Distribution  of  Question  #5  Scores-Answers 


H.  ANALYSIS  OF  QUESTION  #6 

Question  #6  asked:  “What  do  you  see  to  be  the  benefits  associated  with  non- 
compliance  with  this  instruction?” 

Table  16  shows  the  selection  of  scores  that  each  stakeholder  was  able  to  choose 
from  in  answering  the  question,  assigning  a  level  of  benefits  seen  in  not  complying  with 
the  instruction. 
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Choices  of  Score- Answers  for  Question  #6 

Score: 

“  5  ” 

“  4  ” 

“3” 

“2” 

66  55 

Description: 

“many 

significant 

benefits” 

“several 

significant 

benefits” 

“one 

significant 

benefit” 

“a  few 
minor 
benefits” 

“no 

real 

benefit” 

Table  16.  Choices  of  Score- Answers  for  Question  #6 


1.  Purpose  of  Question  #6: 

Question  #6  solicits  the  stakeholder’s  perception  of  any  benefits  associated  with 
not  fully  complying  with  the  COMOPTEVFOR  instruction  and  expects  some  supporting 
examples  for  the  assessment.  The  answer  to  this  question  is  likely  to  supply  some 
evidence  of  the  difficulties  encountered  in  following  the  COMOPTEVFOR  VV&A 
process,  where  non-compliance  produces  some  seeming  benefits. 

2.  Scores  Observed  for  Question  #6: 

Table  17  shows  the  average  score-answers  to  Question  #6.  The  overall  average 
score  is  1 .2,  which  indicates  that  the  typical  stakeholder  surveyed  sees  “no  real  benefit” 
with  instruction  non-compliance.  Those  performing  the  Tester/Accreditor  role  generally 
answered  “no  real  benefit”  but  provided  the  highest  average  score  of  1.4  among  the 
stakeholder  role  categories.  (Thus,  some  of  the  Tester/Accreditors  saw  “a  few  minor 
benefits.”).  The  average  scores  of  the  stakeholder  roles  only  went  down  from  here. 
Program  Managers  came  next,  with  an  average  score  of  1.2,  followed  by  both  the 
Resource  Sponsors  and  the  VV&A  stakeholder  groups,  who  both  posted  the  lowest 
possible  average  scores  of  1.0.  Only  in  the  lower  management  level  of  the  stakeholders, 
who  scored  an  average  of  1 .6,  was  there  any  indication  of  benefit  to  non-compliance. 
This  management  level  average  was  influenced  by  the  2.0  average  of  the 
Tester/Accreditors  within  that  level,  followed  by  Program  Managers  at  1.5,  then 
Resource  Sponsors  at  1.0.  All  middle  and  upper  level  managers  scored  this  question  at 
1.0.  In  sum,  the  scores  indicate  very  strong  stakeholder  agreement  that  no  real  benefits 
are  obtained  from  non-compliance  with  COMOPTEVFOR  5000.1  A. 
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Average  Score- Answers  to  Question  #6 

Stakeholder  Role 

Management  Level  of  Stakeholder 

Avg.  Score 
within  Role 

Lower 

Middle 

Upper 

T  ester/Accreditor 

2.0 

1.0 

1.0 

1.4 

Resource  Sponsor 

1.0 

1.0 

1.0 

1.0 

Program  Manager 

1.5 

1.0 

1.0 

1.2 

VV&A  Standards  Official 

1.0 

1.0 

Avg.  Score  within  Level 

1.6 

1.0 

1.0 

1.2 

Table  17.  Stakeholder  Perceived  Benefits  of  Non-compliance  w/  VV&A  Instruction 

3.  Stakeholder  Comments  for  Question  #6: 

The  stakeholder  comment-answers  signified  the  strongest  accord  of  opinion 
among  the  stakeholders  for  a  given  question.  To  a  person,  no  stakeholder  admitted  of 
any  real  benefit  to  instruction  non-compliance.  Even  those  few  that  indicated  scores  of 
“2”  explained  in  their  comments  that  any  benefit  derived  from  non-compliance  was  really 
a  perversity.  This  comment  from  a  Test/Accreditor  (at  the  lower  management  level)  is  a 
case  in  point: 

From  the  OT  perspective,  there  may  be  (perversely)  one  significant 
benefit  associated  with  non-compliance.  In  the  event  the  PM  has 
not  complied,  he  has  thus  not  justified  his  proposal  to  reduce  the 
physical  testing  required  for  OT.  This  provides  leverage  for  the 
OTD  to  require  more  testing  to  make  up  for  the  lack  of  support  that 
accredited  M&S  provides.  So,  while  non-compliance  may  save 
money  on  M&S  at  the  front  end,  the  PM  ends  up  paying  more  for 
testing  at  the  back  end.  Moreover,  physical  testing  uncovers 
problems  later  when  they  cost  more  to  fix  (if  they  can  be  fixed  at 
that  point),  which  risks  meeting  fewer  requirements. 

This  comment  makes  abundantly  clear  that  any  perceived  “benefit”  of  non-compliance  is 
really  a  liability,  since  the  proponent  will  end  up  spending  more  and  waiting  longer  to 
appropriately  address  the  required  COI  of  OT  and  thus  satisfy  the  Tester/Accreditor.  As 
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a  point  of  departure,  the  issue  of  what  is  meant  by  “non-compliance”  versus  “tailoring”  of 

the  VV&A  process  came  to  light  in  one  of  the  PM  stakeholder’s  comments,  thusly: 

Not  complying  with  COTF  instruction  would  cause  much  more 
pain  than  benefit  (if  any).  However,  PMs  should  find  ways  to 
tailor  the  instruction  to  fit  within  the  constraints  of  their  acquisition 
programs  without  having  a  negative  effect  on  the  integrity  of  the 
VV&A  process. 

The  researchers  noted  from  their  literature  reviews  and  attendance  at  the  VV&A  Forums 
that  tailoring  is  proposed  as  a  technique  that  enables  PMs  to  individually  comply  with  the 
prescribed  VV&A  process  in  accordance  with  the  particular  circumstances  /  constraints 
of  their  programs.  However,  no  further  interview  comments  were  provided  in  this  area. 


4.  Score  /  Comment  Associations  for  Question  #6: 

The  distribution  of  individual  stakeholder  score-answers  shown  in  Table  18  shows 
that  all  of  the  scores  lay  at  the  bottom  of  the  assessment  of  risk  scale.  12  of  14 
stakeholders  said  that  there  was  “no  real  benefit”  with  instruction  non-compliance.  The 
remaining  two  are  distributed  at  the  “2”  and  “3”  score  levels,  principally  to  admit  that 
some  “perverse”  benefits  are  derived  but,  as  discussed  above,  are  really  bigger  liabilities. 
The  idea  of  “tailoring”  as  a  mechanism  for  PMs  to  vary  their  approach  to  VV&A  process 
compliance  was  raised  but  did  not  go  further  in  discussion. 


5.  Conclusions  for  Question  #6: 

The  idea  of  reaping  any  benefit  through  non-compliance  with  COMOPTEVFOR- 
INST  5000.1  A  was  really  a  “non-starter”  for  most  of  the  stakeholders  interviewed.  One 
stakeholder  opined,  “This  question  doesn’t  really  make  sense,  because  non-compliance 
with  instruction  brings  no  benefit  whatsoever.”  Nevertheless,  asking  the  question  was 
useful,  as  it  generated  some  interesting  comments  on  the  adverse  impacts  of  non- 
compliance.  These  included:  (a)  “M&S  activities  are  not  coordinated  with  acquisition 
strategy;”  (b)  “T&E  strategy  does  not  provide  validation  data;”  (c)  “Models  are  not 
accredited  prior  to  use;”  and  (d)  “M&S  use  and  VV&A  strategy  are  not  in  the  TEMP.” 
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The  question  also  raised  the  issue  of  “tailoring”  the  VV&A  process  to  accommodate  the 
individual  constraints  of  the  various  surface  ship  acquisition  programs. 


Distribution  of  Question  #6  Score -Answers 

Level  of  Benefits  Perceived  with  Non-Compliance 

Count 

Stakeholders  who  provided  a  score  of  “5” 

0 

Stakeholders  who  provided  a  score  of  “4” 

0 

Stakeholders  who  provided  a  score  of  “3” 

1 

Stakeholders  who  provided  a  score  of  “2” 

1 

Stakeholders  who  provided  a  score  of  “1” 

12 

Total 

14 

Table  18.  Distribution  of  Question  #6  Scores-Answers 


I.  ANALYSIS  OF  QUESTION  #7 

Question  #  7  asked:  “What  do  you  see  to  be  the  costs  associated  with  non- 
compliance  with  this  instruction?” 

Table  19  shows  the  selection  of  scores  that  each  stakeholder  was  able  to  choose 
from  in  answering  the  question,  assigning  a  level  of  costs  seen  in  not  complying  with  the 
instruction. 


Choices  of  Score- Answers  for  Question  #7 

Score: 

“5  ” 

“  4  ” 

“3” 

“2” 

U  ^  55 

“many 

“several 

“one 

“a  few 

“no 

Description: 

significant 

significant 

significant 

minor 

real 

costs” 

costs” 

cost” 

costs” 

cost” 

Table  19.  Choices  of  Score- Answers  for  Question  #7 
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1.  Purpose  of  Question  #7: 

Question  #7  solicits  the  stakeholder’s  perception  of  the  costs  associated  with  not 
complying  with  the  COMOPTEVFOR  instruction  and  expects  some  supporting  examples 
for  the  assessment.  The  answer  to  this  question  is  likely  to  point  to  momentous  financial 
dangers  of  non-compliance,  which  ultimately  must  be  controlled  by  the  M&S  proponent 
and  reconciled  by  the  sponsor. 

2.  Scores  Observed  for  Question  #7: 

Table  20  shows  the  average  score-answers  to  Question  #7.  The  overall  average 
score  is  4.0,  which  indicates  that  the  typical  stakeholder  surveyed  sees  “several 
significant  costs”  incurred  when  not  complying  with  the  instruction.  Those  perfonning 
the  Tester/Accreditor  role  generally  answered  “many  significant  costs”  and  scored  a  4.6 
average,  the  highest  score  among  the  stakeholder  role  categories.  Next  in  score  order  was 
the  VV&A  Standards  Official  role,  who  provided  a  score  of  4,  which  exactly  matched  the 
overall  stakeholder  average,  likewise  indicating  “several  significant  costs.”  Those 
performing  the  Resource  Sponsor  role  scored  a  3.7  average  (just  0.3  below  the  overall 
average),  also  indicating  that  “several  significant  costs”  would  be  suffered  through  non- 
compliance  with  the  instruction.  The  Program  Managers  scored  an  average  of  3.6,  which 
is  still  just  0.4  below  the  overall  average  and  describes  the  same  cost  impact.  The 
average  scores  within  the  lower,  middle,  and  upper  management  levels  are  4.6,  4.0,  and 
3.4  respectively.  These  scores  indicate  that  stakeholders  at  the  three  management  levels 
vary  +/-  0.6  around  the  mean  of  “several  significant  costs”  regarding  non-compliance 
with  the  instruction.  Thus,  the  relatively  small  variance  of  stakeholder  role  and 
management  level  average  scores  indicates  a  strong  consensus  of  all  stakeholders  that  the 
costs  of  non-compliance  are  both  “significant”  and  “several.” 

3.  Stakeholder  Comments  for  Question  #7: 

The  stakeholders’  comment- answers  indicated  that  a  strong  consensus  exists 
regarding  the  high  costs  of  non-compliance  with  the  VV&A  instruction.  The  keynote  of 
this  hannony  was  heard  from  the  very  first  Tester/Accreditor  interviewed,  as  follows: 
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The  proponent  cannot  perfonn  adequately  the  needed  design 
studies,  trade  studies,  pre-test  planning,  pre-test  predictions,  and 
post-test  analysis  when  the  M&S  is  of  poor  quality,  as  would  be 
indicated  by  its  failure  to  be  accredited.  One  cannot  trust  the 
results!  Consequently,  it  is  harder  to  analyze  and  solve  problems 
early.  This  failure  would  be  avoided  if  sufficient  funding  were 
applied  to  M&S  development  and  its  supporting  VV&A.  While 
the  costs  associated  with  non-compliance  are  sometimes  hard  to 
quantify,  the  effects  are  nonetheless  apparent. 


Average  Score- Answers  to  Question  #7 

Stakeholder  Role 

Management  Level  of  Stakeholder 

Avg.  Score 
within  Role 

Lower 

Middle 

Upper 

T  ester/Accreditor 

5.0 

5.0 

4.0 

4.6 

Resource  Sponsor 

4.0 

3.0 

4.0 

3.7 

Program  Manager 

4.5 

4.0 

1.0 

3.6 

VV&A  Standards  Official 

4.0 

4.0 

Avg.  Score  within  Level 

4.6 

4.0 

3.4 

4.0 

Table  20.  Stakeholder  Perceived  Costs  of  Non-compliance  with  VV&A  Instruction 


The  other  stakeholders  sang  essentially  the  same  song  in  their  comments.  One 

particularly  strong  solo,  which  offered  a  stark  example  of  a  cost  consequence,  was  given 

by  a  Resource  Sponsor,  as  follows: 

In  the  case  of  one  ship  acquisition  program,  it  cost  $250M  (which 
had  to  be  taken  from  other  program  funds)  to  perform  the  physical 
operational  test  that  the  M&S  sought  to  accomplish  in  a  virtual 
environment.  Perhaps  earlier  PM  interaction  with  the  Tester/ 
Accreditor  and  the  collection  of  some  additional  physical  (real) 
data  would  have  led  to  a  credible  M&S  solution. 


Program  Managers,  too,  cited  significant  financial  impacts  for  non-compliance.  This 

thoughtful  comment  particularly  resonated: 

Not  following  the  VV&A  instruction  creates  a  significant 
probability  of  incurring  added  costs,  especially  when  trying  to 
validate  and  verify  the  models  of  federations.  On  what  basis  will 
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PMs  be  confident  that  the  ship  systems  they  are  delivering  will 
perforin  the  way  they  are  predicting?  Physical  testing  will  have  to 
be  added  to  prove  the  VV&A  case.  This  is  necessary  because  no 
one  can  buy  or  accept  a  product  of  which  they  are  unsure, 
especially  when  the  stakes  are  high.  The  costs  are  ultimately  borne 
by  the  warfighter. 


Finally,  the  VV&A  Standards  Official  role  voiced  this  final  note:  “Non-compliance  with 
the  instruction  would  continue  to  cost  the  U.S.  Navy  more  money.” 


4.  Score  /  Comment  Associations  for  Question  #7: 

Table  21  shows  from  its  distribution  of  raw  scores  that  most  (11  out  of  14) 
stakeholders  believe  the  costs  incurred  by  not  complying  with  COMOPTEVFORINST 
5000.1  A  to  be  (at  least)  “several  significant  costs”  (with  scores  of  “4”  and  “5”).  Two 
stakeholders  saw  at  least  “one  significant  cost”  and,  because  of  their  attribution  of 
“significant”  with  regard  to  costs  experienced,  should  also  be  grouped  with  the  super 
majority.  Only  one  (lonely)  dissenter  is  evident.  This  member  of  the  PM  group  pointed 
out  that,  “There  is  no  real  cost  associated  with  non-compliance  if  VV&A  is  not  done. 
However,  the  cost  to  go  back  later  and  conduct  VV&A  will  be  much  more  expensive.” 
Thus,  the  one  dissenting  opinion  is  really  saying  (perversely)  that  non-compliance  cost 
nothing  if  one  chooses  not  to  participate. 


Distribution  of  Question  #7  Score -Answers 

Level  of  Costs  Perceived  with  Non-Compliance 

Count 

Stakeholders  who  provided  a  score  of  “5” 

5 

Stakeholders  who  provided  a  score  of  “4” 

6 

Stakeholders  who  provided  a  score  of  “3” 

2 

Stakeholders  who  provided  a  score  of  “2” 

0 

Stakeholders  who  provided  a  score  of  “1” 

1 

Total 

14 

Table  2 1 .  Distribution  of  Question  #7  Scores-Answers 
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5.  Conclusions  for  Question  #7: 

The  vast  majority  of  stakeholders  interviewed  consider  the  costs  of  not  complying 
with  the  COMOPTEVFOR  VV&A  instruction  to  be  (at  least)  “several”  and  “significant.” 
A  number  of  concrete  (i.e.,  real  dollar  value)  examples  were  provided.  Regardless  of 
stakeholder  role  or  management  level  in  their  respective  organizations,  all  participants 
appreciated  the  potential  cost  increases  to  U.S.  Navy  surface  ship  acquisition  programs  if 
COMOPTEVFORINST  5000.1  A  were  not  followed  in  support  of  M&S  for  OT 
accreditation. 

J.  ANALYSIS  OF  QUESTION  #8 

Question  #8  asks:  “What  do  you  see  to  be  the  risks  associated  with  non- 
compliance  with  this  instruction?” 

Table  22  shows  the  selection  of  scores  that  each  stakeholder  was  able  to  choose 
from  in  answering  the  question,  assigning  a  level  of  risks  seen  in  not  complying  with  the 
instruction. 


1.  Purpose  of  Question  #8: 

Question  #8  solicits  the  stakeholder’s  perception  of  the  risks  associated  with  not 
fully  complying  with  the  COMOPTEVFOR  instruction  and  expects  some  supporting 
examples  for  the  assessment.  The  answer  to  this  question  is  likely  to  point  out  the 
hazards  of  deviating  from  the  standards  of  VV&A  practice  put  forth  by  the  instruction 
and  remind  practitioners  that  the  ultimate  accreditation  of  the  proposed  M&S  is  at  stake. 


Choices  of  Score- Answers  for  Question  #8 

Score: 

“5” 

“  4  ” 

“3” 

“2  ” 

“many 

“several 

“one 

“a  few 

“no 

Description: 

significant 

significant 

significant 

minor 

real 

risks” 

risks” 

risk” 

risks” 

risk” 

Table  22.  Choices  of  Score-Answers  for  Question  #8 
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2.  Scores  Observed  for  Question  #8: 

Table  23  shows  the  average  score-answers  to  Question  #8.  The  overall  average 
score  is  4.4,  which  indicates  that  the  typical  stakeholder  surveyed  sees  (at  least)  “several 
significant  risks”  incurred  when  not  complying  with  the  instruction.  Those  perfonning 
the  Program  Manager  generally  answered  “many  significant  risks”  and  scored  a  4.8 
average,  the  highest  score  among  the  stakeholder  role  categories.  Next  in  score  order  was 
the  Tester/Accreditor  role,  who  provided  an  average  score  of  4.4,  which  exactly  matched 
the  overall  stakeholder  average,  likewise  indicating  (at  least)  “several  significant  risks.” 
The  VV&A  Standards  Official  role  scored  4.0,  which  is  still  just  0.4  below  the  overall 
average  and  describes  the  same  cost  impact.  Those  performing  the  Resource  Sponsor 
role  scored  a  3.7  average  (0.7  below  the  overall  average),  which  closely  approaches  the 
indication  of  “several  significant  risks”  that  would  be  suffered  through  non-compliance 
with  the  instruction.  The  average  scores  within  the  lower,  middle,  and  upper 
management  levels  are  4.4,  4.5,  and  4.2  respectively.  These  scores  indicate  that 
stakeholders  at  the  three  management  levels  vary  +/-  0.3  (maximum)  around  the  mean  of 
“several  significant  risks”  regarding  non-compliance  with  the  instruction.  Thus,  the 
relatively  small  variance  of  stakeholder  role  and  management  level  average  scores 
indicates  a  strong  consensus  among  the  stakeholders  that  non-compliance  entails  “several 
significant  risks.” 


Average  Score- Answers  to  Question  #8 

Stakeholder  Role 

Management  Level  of  Stakeholder 

Avg.  Score 
within  Role 

Lower 

Middle 

Upper 

T  ester/Accreditor 

4.5 

5.0 

4.0 

4.4 

Resource  Sponsor 

4.0 

3.0 

4.0 

3.7 

Program  Manager 

4.5 

5.0 

5.0 

4.8 

VV&A  Standards  Official 

4.0 

4.0 

Avg.  Score  within  Level 

4.4 

4.5 

4.2 

4.4 

Table  23.  Stakeholder  Perceived  Risks  of  Non-compliance  with  VV&A  Instruction 
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3.  Stakeholder  Comments  for  Question  #8: 

The  stakeholders’  comment- answers  indicated  that  a  strong  consensus  exists 

regarding  the  high  risks  of  non-compliance  with  the  VV&A  instruction.  The  comments 

that  identify  the  risks  entailed  support  the  sense  of  “real,”  “varied,”  and  “significant.” 

The  main  point  raised  was  captured  in  this  stakeholder  statement,  “The  risks  are  that  the 

requirements  of  the  M&S  will  not  be  met  and  therefore  the  OT&E  test  requirements 

would  not  be  fully  satisfied.”  Another  Tester/Accreditor  offered  this  “real  life”  example: 

One  significant  risk  associated  with  non-compliance  was  recently 
made  evident  where  a  ship  acquisition  program  was  held  up  at  MS 
B  while  resolving  the  issue  of  whether  the  use  of  M&S  would 
satisfy  one  of  the  OT  requirements. 


A  further  case-in-point  was  made  by  a  Resource  Sponsor,  as  follows: 

Several  significant  risks  are  associated  with  non-compliant  VV&A, 
including  the  need  to  “re-synchronize”  or  “re-coordinate”  the 
acquisition  program’s  “tracks”  (i.e.,  the  kind  of  activities/events 
identified  in  the  TEMP).  As  this  would  translate  to  cost  increases, 
the  PM  would  have  to  go  back  to  the  Resource  Sponsor  for  help. 
But  the  Resource  Sponsor’s  solution  is  usually  to  advise  the  PM  to 
“take  it  out  of  hide”  and  reprogram  existing  funds  to  support  the 
changes. 


Finally,  a  PM  provided  this  discussion  on  the  risks  of  getting  “false-positives,”  to  wit: 

Getting  “false-positives”  are  the  risks  associated  with  non- 
compliance.  It  can  be  presumed  that  the  PMs  are  moving  forward 
in  good  faith  to  conduct  VV&A  on  their  M&S.  However,  when 
they  do  not  use  the  COTF  prescribed  process,  they  end  up  having 
to  develop  their  own.  This  will  result  in  unevenness  of  results 
across  the  PMs.  Some  PMs  will  be  more  stringent  than  COTF,  but 
most  will  likely  be  less  stringent.  Those  who  are  less  stringent  run 
the  risk  of  getting  false-positive  results,  that  is,  of  qualifying  an 
M&S  that  is  really  deficient.  The  effects  of  that  false  positive  will 
be  rolled-up  into  the  system  to  which  the  M&S  is  applied.  The 
resulting  disconnect  between  expected  perfonnance  and  actual 
performance  means  “bad  news”  for  the  fleet. 


Many  other  observations  and  examples  were  cited.  The  VV&A  Standards  Official  role 
(once  again)  made  an  excellent  closing  comment  on  the  issue  by  stating: 
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There  are  several  significant  risks  associated  with  non-compliance; 
the  foremost  among  them  is  the  lack  of  credibility  for  those  models 
developed  without  regard  to  compliance. 

4.  Score  /  Comment  Associations  for  Question  #8: 

Table  24  shows  from  its  distribution  of  raw  scores  that  virtually  all  of  the 
stakeholders  interviewed  believe  that,  by  not  complying  with  COMOPTEVFORINST 
5000.1  A,  M&S  proponents  would  lay  themselves  open  to  (at  least)  “several  significant 
risks”  (thereby  giving  scores  of  “4”  and  “5”).  The  remaining  stakeholder  still  thinks  that 
the  incurred  risk  is  significant  (but  just  cited  one  example).  Other  than  the  various 
examples  of  risk  cited  in  their  comments,  many  of  the  stakeholders  noted  that  risks,  when 
instantiated,  most  often  translate  into  added  program  costs. 


Distribution  of  Question  #8  Score -Answers 

Level  of  Risks  Perceived  with  Non-Compliance 

Count 

Stakeholders  who  provided  a  score  of  “5” 

6 

Stakeholders  who  provided  a  score  of  “4” 

7 

Stakeholders  who  provided  a  score  of  “3” 

1 

Stakeholders  who  provided  a  score  of  “2” 

0 

Stakeholders  who  provided  a  score  of  “1” 

0 

Total 

14 

Table  24.  Distribution  of  Question  #8  Scores-Answers 


5.  Conclusions  for  Question  #8: 

An  overwhelming  majority  of  the  stakeholders  view  the  risks  of  not  complying 
with  the  COMOPTEVFOR  VV&A  instruction  to  be  (at  least)  “several”  and  “significant.” 
A  number  of  “real  life”  examples  were  provided,  which  demonstrated  the  seriousness  of 
the  risks  that  non-compliant  M&S  proponents  would  take  on.  Risks  realized  are  often 
“felt”  in  the  form  of  added  costs  and  extended  schedules.  The  consensus  demonstrated 
by  the  stakeholders  signals  that  COMOPTEVFORINST  5000.1  A  is  effectively  educating 


92 


the  diverse  U.S.  Navy  surface  ship  acquisition  participants  on  the  importance  of 
following  the  standards  of  the  VV&A  process. 

K.  ANALYSIS  OF  QUESTION  #9 

Question  #  9  asks:  “To  what  extent  did  the  change  from  the  superseded 
COMOPTEVFORINST  5000.1  (of  5  September  1995)  to  the  new  COMOPTEVFOR- 
INST  5000.1  A  (of  9  September  2004)  improve  or  degrade  the  VV&A  of  M&S  used  for 
OT&E?” 

Table  25  shows  the  selection  of  scores  that  each  stakeholder  was  able  to  choose 
from  in  answering  the  question,  assigning  a  level  of  improvement  (or  degradation)  seen 
in  changing  to  the  new  instruction. 


1.  Purpose  of  Question  #9: 

Question  #9  is  posed  to  elicit  comment  from  the  stakeholder  that  recognizes 
whether  any  improvements  (or  degradations)  were  obtained  in  replacing  the  old 
COMOPTEVFOR  VV&A  instruction  with  the  new  (current)  instruction.  This  may  not 
be  an  easy  question  to  answer,  as  the  stakeholder  is  expected  to  be  sufficiently 
experienced  with  (or  knowledgeable  of)  the  letter,  spirit,  and  practice  of  both  instructions. 


Choices  of  Score- Answers  for  Question  #9 

Score: 

“5” 

“  4  ” 

“3  ” 

“2  ” 

66  55 

Description: 

“significant 

improve¬ 

ments” 

“minor 

improve¬ 

ments” 

“no  real 
improve¬ 
ments” 

“minor 

degrada¬ 

tions” 

“significant 

degrada¬ 

tions” 

Table  25.  Choices  of  Score- Answers  for  Question  #9 
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2.  Scores  Observed  for  Question  #9: 

Table  26  shows  the  average  score-answers  to  Question  #9.  The  overall  average 
score  is  3.9,  which  indicates  that  the  typical  stakeholder  surveyed  saw  “minor 
improvements”  in  going  to  the  new  the  instruction. 


Average  Score- Answers  to  Question  #9 

Stakeholder  Role 

Management  Level  of  Stakeholder 

Avg.  Score 
within  Role 

Lower 

Middle 

Upper 

T  ester/Accreditor 

3.0 

5.0 

5.0 

4.2 

Resource  Sponsor 

5.0 

3.0 

3.0 

3.7 

Program  Manager 

3.5 

3.5 

4.0 

3.6 

VV&A  Standards  Official 

4.0 

4.0 

Avg.  Score  within  Level 

3.6 

3.8 

4.2 

3.9 

Table  26.  Stakeholder  Assessment  of  Improvement  Gained  with  New  Instruction 

Those  performing  the  Tester/  Accreditor  role  generally  answered  “minor 
improvements”  and  scored  a  4.2  average,  the  highest  score  among  the  stakeholder  role 
categories.  Next  in  order  of  score  is  the  VV&A  Standards  Official  role,  who  provided  a 
score  of  4,  which  is  only  0.1  above  the  overall  stakeholder  average,  indicating  that  he/she 
thinks  that  “minor  improvements”  were  achieved  by  the  new  instruction.  Then  the 
Resource  Sponsor  and  the  Program  Manager  roles  follow  with  average  scores  of  3.7  and 
3.6  (which  are  0.2  and  0.3  below  the  overall  average),  respectively.  These  scores  also 
indicate  that  the  stakeholders  saw  “minor  improvements”  in  the  new  instruction.  The 
average  scores  within  the  lower,  middle,  and  upper  management  levels  are  3.6,  3.8,  and 
4.2,  respectively,  which  show  a  modest  upward  trend  by  the  ascending  levels  of 
management  in  grading  the  levels  of  improvement  found  in  the  new  instruction. 
Nevertheless,  the  scores  indicate  that  all  three  management  levels  are  saying  “minor 
improvements”  were  achieved  in  moving  from  the  old  to  the  new  instruction. 
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3.  Stakeholder  Comments  for  Question  #9: 

The  stakeholders’  comment-answers  indicated  that  no  consensus  exists  regarding 
the  recognition  of  VV&A  process  improvements  (or  degradations)  in  moving  from  the 
old  COMOPTEVFORINST  5000.1  to  the  new  COMOPTEVFORINST  5000.1  A.  Rather, 
the  comments  indicated  that  three  perspectives  are  maintained:  (a)  No  real  improvement 
was  achieved;  (b)  minor  improvements  were  achieved;  and  (c)  significant  improvements 
were  achieved.  In  the  first  case,  a  plurality  of  six  (of  the  14)  stakeholders  commented 
that  no  real  improvements  were  evident  in  the  instruction  revision.  The  general  remark 
of  this  group  was,  “The  jury  is  still  out.”  They  observed  that  the  new  instruction  has  not 
been  in  place  long  enough  to  collect  data  and  perform  a  genuine  assessment  of  its 
performance.  Now  the  second  group  (numbering  four  stakeholders)  has  apparently  seen 
some  data  that  leads  their  members  to  conclude  that  “minor  improvements”  were 
achieved.  Three  points  were  made  in  support  of  this  assessment: 

•  The  new  instruction  is  more  user-friendly  and  shorter. 

•  The  new  instruction  provides  a  sense  of  structure,  format  and  process.  [It] 
guides  the  PM  through  the  steps  of  the  VV&A  process  with  only  minor  difficulties. 

•  The  new  instruction  better  articulates  accepted  VV&A  standards  and 
provides  more  specific  guidance  than  the  general  philosophy  of  the  old  instruction. 

Finally,  the  third  group  (also  numbering  four  members)  sees  evidence  of  “significant 
improvement”  in  the  move  to  the  new  instruction.  Observations,  such  as  these,  were 
offered  in  support  of  this  group’s  view: 

•  With  the  old  instruction,  PMs  did  not  know  what  the  requirements  were. 

•  [The  new  instruction]  provides  a  far  better  roadmap  than  before. 

•  The  earlier  the  OTD  is  involved,  the  better  the  compliance.  The  new 
instruction  better  promotes  this. 

•  The  changes  incorporated  in  the  new  instruction  are  “golden,”  as  they 
allow  the  PM  more  flexibility  while  still  expecting  him  to  interact  with  COTF  early  in  the 
process  for  delivering  VV&A  products. 
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Hence,  the  interviewed  stakeholders’  statements  demonstrate  a  three-way  split  in  the 
community  of  VV&A  participants  regarding  the  level  of  improvement  achieved  by  the 
recent  revision  of  the  COMOPTEYFOR  instruction. 


4.  Score  /  Comment  Associations  for  Question  #9: 

Table  27  shows  from  its  distribution  of  raw  scores  that  a  three-way  split  exists 
regarding  stakeholder  opinion  on  the  level  of  improvement  gained  in  going  to  the  new 
instruction,  COMOPTEVFORINST  5000.1  A  of  9  September  2004.  The  largest  of  the 
groups  (with  6  members)  graded  the  new  instruction  at  “3,”  which  indicated  their 
judgment  that  it  caused  (so  far)  “no  real  improvements”  in  the  VV&A  process  practiced. 


Distribution  of  Question  #9  Score -Answers 

Level  of  Improvement  Gained  with  New  Instruction 

Count 

Stakeholders  who  provided  a  score  of  “5” 

4 

Stakeholders  who  provided  a  score  of  “4” 

4 

Stakeholders  who  provided  a  score  of  “3” 

6 

Stakeholders  who  provided  a  score  of  “2” 

0 

Stakeholders  who  provided  a  score  of  “1” 

0 

Total 

14 

Table  27.  Distribution  of  Question  #9  Scores-Answers 


The  remaining  two  groups  comprised  four  members  each,  where  the  second  camp  saw 

“minor  improvements”  and  the  third  cited  “significant  improvements.”  These  two 

factions  each  provided  several  observations  to  support  their  assessments.  One  score  / 

comment  association  stood  out  in  particular,  indicating  a  shift  in  VV&A  process  focus: 

[A  significant  improvement]  includes  the  insight  that  the 
participants  (within  the  established  accreditation  process)  should 
move  from  a  focus  on  the  fidelity  of  the  model  to  the 
appropriateness  of  the  model  for  its  intended  use.  The  level  of 
resolution  to  be  achieved  for  the  model  depends  on  the  questions 
being  asked. 
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Since  the  majority  of  the  stakeholders’  W&A  experiences  were  under  the  old 
instruction,  the  researchers  found  the  answers  to  this  question  illuminating. 


5.  Conclusions  for  Question  #9: 

With  an  overall  stakeholder  average  score  of  3.9,  but  showing  a  three-way 
divergence  of  raw  score-answers  (as  was  also  evident  in  the  comment-answers),  the 
answer  to  the  question  of,  “Whether  improvement  was  obtained  by  the  new  instruction?” 
boils  down  to  the  realization:  “It  depends.”  The  experience  of  the  stakeholders  varied  on 
this  issue  but  did  not  depend  on  the  stakeholder  role  or  management  level.  Therefore,  it 
appears  that  the  issue  varies  for  the  moment  on  the  experiences  of  the  individuals. 

L.  ANALYSIS  OF  QUESTION  #10 

Question  #10  asks:  “What  improvements  would  you  recommend  for  COMOP- 
TEVFOR’s  Accreditation  process  (beyond  that  prescribed  by  the  new  COMOPTEVFOR- 
INST  5000.1  A)? 

Table  28  shows  the  selection  of  scores  that  each  stakeholder  was  able  to  choose 
from  in  answering  the  question,  assigning  a  level  of  opportunity  for  improvement  beyond 
that  achieved  by  the  new  instruction. 

1.  Purpose  of  Question  #10: 

Question  #10  is  posed  to  elicit  recommendations  for  improvements  to  any  future 
edition  of  the  COMOPTEVFOR  VV&A  instruction  or  for  improvements  to  the  U.S. 
Navy’s  VV&A  process  in  general.  While  the  answer  to  this  question  is  speculative,  it 
nevertheless  provides  an  outlet  for  expressing  solutions  to  the  difficulties  (regrettable 
costs  and  risks)  still  found  in  complying  with  the  current  COMOPTEVFOR  VV&A 
instruction. 
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Choices  of  Score- Answers  for  Question  #10 

Score: 

“5” 

“  4  ” 

“3” 

“2” 

tt  ^  59 

“many 

“several 

“one 

“a  few 

“no 

Description: 

significant 

significant 

significant 

minor 

real 

improve- 

improve- 

improve- 

improve- 

improve- 

ments” 

ments” 

ment” 

ments” 

ment” 

Table  28.  Choices  of  Score-Answers  for  Question  #10 

2.  Scores  Observed  for  Question  #10: 

Table  29  shows  the  average  score-answers  to  Question  #10.  The  overall  average 
score  is  2.9,  which  indicates  that  the  typical  stakeholder  surveyed  saw  an  opportunity  to 
make  “one  significant  improvement”  beyond  that  which  is  currently  described  in 
COMOPTEVFORINST  5000.1  A.  Those  performing  the  Tester/Accreditor  role 

generally  answered  “one  significant  improvement”  and  scored  a  3.2  average,  the  highest 
score  among  the  stakeholder  role  categories.  Next  in  order  of  score  is  the  Resource 
Sponsor  role,  who  provided  an  average  score  of  3.0,  which  is  just  0.1  above  the  overall 
stakeholder  average,  indicating  that  this  group  thinks  that  “one  significant  improvement” 
can  be  made  to  the  5000. 1 A  instruction.  Closely  following  is  the  Program  Manager  role, 
with  an  average  score  of  2.8  (just  0.1  below  the  overall  average),  likewise  indicating  that 
these  stakeholders  saw  the  opportunity  for  “one  significant  improvement”  in  the  5000.1  A 
instruction.  Finally,  well  down  in  the  score  order,  is  the  W&A  Standards  Official  role, 
who  provided  a  score  of  1,  which  indicates  that  he/she  saw  “no  [opportunity  for]  real 
improvement.”  The  average  scores  within  the  lower,  middle,  and  upper  management 
levels  are  3.0,  2.3,  and  3.2,  respectively,  which  shows  a  one  point  downward  dip  in  the 
score  trend  at  the  middle  management  level.  This  indicates  that  mid-management,  on 
average,  only  sees  opportunities  for  “a  few  minor  improvements,”  while  the  lower  and 
upper  levels  see  “one  significant  improvement,”  which  closely  corresponds  to  the  overall 
stakeholder  average. 
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Average  Score- Answers  to  Question  #10 

Stakeholder  Role 

Management  Level  of  Stakeholder 

Avg.  Score 
within  Role 

Lower 

Middle 

Upper 

T  ester/Accreditor 

4.5 

1.0 

3.0 

3.2 

Resource  Sponsor 

1.0 

3.0 

5.0 

3.0 

Program  Manager 

2.5 

2.5 

4.0 

2.8 

VV&A  Standards  Official 

1.0 

1.0 

Avg.  Score  within  Level 

3.0 

2.3 

3.2 

2.9 

Table  29.  Perceived  Opportunity  for  Improvement  of  COTF’s  Accreditation  Process 


3.  Stakeholder  Comments  for  Question  #10: 

The  stakeholders’  comment-answers  indicated  that  no  consensus  exists  regarding 

the  opportunity  for  making  improvements  to  the  VV&A  process  described  in  the  new 

COMOPTEVFORINST  5000.1  A.  Rather,  the  comments  indicated  that  there  are  two 

opposing  camps  on  the  issue  (and  possibly  a  third  one  somewhere  in  the  middle).  The 

two  opposing  perspectives  are:  (a)  There  is  no  real  opportunity  (or  impetus)  for 

improvement;  and  (b)  There  is  opportunity  for  making  significant  improvements  to  the 

latest  instruction.  Six  (of  the  14)  stakeholders  make  up  the  first  camp  and  either 

recommended  “no  real  improvement”  or  only  “a  few  minor  improvements”  regarding 

COMOPTEVFORINST  5000.1  A.  An  identically  sized  (six  of  14)  second  camp 

recommended  either  “many”  or  “several  significant  improvements”  to  the  instruction. 

There  are  two  stakeholders  in  the  middle  of  the  debate  who  see  opportunity  for  only  “one 

significant  improvement.”  However,  this  “middle  ground”  can  really  be  assigned  to 

Camp  #2  (who  recommend  more  than  one  significant  improvement)  because  the  single 

improvement  they  are  recommending  is  still  “significant,”  as  opposed  to  minor.  The 

below  comment  from  a  Tester/Accreditor  expresses  the  second  camp’s  view  succinctly: 

Since  OT  is  the  final  step  before  full  production,  it  would  be  most 
efficient  to  offer  the  PM  a  VV&A  instruction  that  contains  both 
policy  and  detailed  “how  to”  guidance  a  single,  usable  resource. 

The  PM  would  there  find  efficient  methods  to  organize,  estimate 
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costs,  determine  schedules,  and  finish  a  quality  M&S  on  time  and 
within  budget. 

A  more  detailed  comment  from  second  Tester/Accreditor  addresses  the  need  to  revise  the 

instruction  to  better  support  the  reuse  of  M&S: 

Among  the  possible  improvements  to  be  made  is  finding  a  way  to 
make  models  more  available  for  reuse  -  both  within  the  U.S.  Navy 
and  among  all  the  Military  Services.  Testing  in  the  Joint  Force 
Environment  relies  on  a  having  an  adequately  modeled  virtual 
environment.  More  funding  is  needed  to  fully  support  this 
initiative,  along  with  a  technical  commitment  at  and  above  any 
particular  Service’s  ability.  Then  the  question  can  be  asked,  “What 
does  a  PM  expect  to  be  able  to  do  when  he/she  wants  to  reuse  the 
M&S  developed  by  another,  only  to  find  that  this  PM  is  not 
following  the  Service’s  VV&A  process?” 


The  opposing  camp  offered  comments  like  this  one: 

There  are  no  real  improvements  that  can  be  made  to  COTF’s 
Accreditation  process.  The  key  issue  for  improving  the 
accreditation  process  regards  the  implementation  of  VV&A  and 
not  necessarily  what  the  VV&A  requires  by  way  of  procedure. 
Instead  of  focusing  on  the  process  for  producing  documentation, 
the  participants  should  focus  on  the  technical  content  conveyed  in 
the  documentation.  In  other  words,  the  instruction  should  not 
emphasize  how  to  do  conduct  the  W&A  process  or  require  close 
adherence  to  a  checklist.  Rather,  the  focus  should  be  on  the 
appropriateness  of  the  model  to  answer  the  technical  question 
related  to  its  intended  use. 


The  comment-answers  generated  by  this  question  were  the  most  varied  and  extensive  of 
those  collected  by  the  interviews.  The  reader  is  encouraged  to  review  the  Appendix  for 
the  remaining  comments  that  provide  specific  recommendations  for  improvements,  or 
reasons  why  improvements  to  COMOPTEFORINST  5000.1  A  are  not  warranted. 


4.  Score  /  Comment  Associations  for  Question  #10: 

Table  30  shows  from  its  distribution  of  raw  scores  that  the  two  highest  score 
counts  (at  4)  are  for  the  scores  of  “4”  and  “1,”  which  represent  the  polar  opposite 
recommendations  of  “several  significant  improvements”  and  “no  real  improvement.” 
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The  remaining  counts  are  half  (at  2)  of  those  assigned  to  each  of  the  two  poles.  These 
suggest  the  down  steps  on  either  side  of  the  two  primary  opinions. 


Distribution  of  Question  #10  Score- Answers 

Level  of  Opportunity  for  VV&A  Improvement 

Count 

Stakeholders  who  provided  a  score  of  “5” 

2 

Stakeholders  who  provided  a  score  of  “4” 

4 

Stakeholders  who  provided  a  score  of  “3” 

2 

Stakeholders  who  provided  a  score  of  “2” 

2 

Stakeholders  who  provided  a  score  of  “1” 

4 

Total 

14 

Table  30.  Distribution  of  Question  #10  Scores-Answers 


5.  Conclusions  for  Question  #10: 

The  overall  stakeholder  average  score  of  2.9  masks  the  presence  of  two  distinct 
camps  of  opinion  regarding  the  opportunity  to  make  further  improvements  to  COM- 
OPTEVFOR’s  VV&A  instruction.  The  first  camp  does  not  see  the  need  to  improve  the 
instruction,  although  they  generally  advise  other  improvements  (such  as  making  better 
efforts  to  follow  the  instruction  “as  is”).  On  the  other  hand,  the  second  camp  sees  plenty 
opportunity  to  make  improvements  to  the  instruction,  mostly  because  they  believe  a 
revised  instruction  can  provide  more  “how  to”  guidance.  Despite  how  well  VV&A  is 
doing  under  the  current  instruction,  a  simple  majority  (8  of  the  14  interviewed)  sees  the 
need  to  make  at  least  “one  significant  improvement”  to  COMOPTEVFORINST  5000.1  A. 


M,  CONCLUSION 

This  chapter  analyzed  the  answers  supplied  for  the  ten  questions  administered  to 
U.S.  Navy  Surface  Ship  VV&A  stakeholders  through  semi-structured  interviews.  The 
data  was  sorted,  averaged,  and  grouped  to  identify  areas  of  consensus  or  divergence. 
Stakeholder  comments  were  highlighted  to  assist  the  reader’s  understanding  of  the  issues 
raised  and  to  present  options  for  consideration.  A  conclusion  was  reached  for  each 
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question’s  concern.  This  will  provide  the  basis  for  making  the  global  conclusions  and 
recommendations  that  are  offered  in  Chapter  IV. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

This  chapter  provides  conclusions  that  address  the  thesis  objectives  and  answer 
the  research  questions  that  were  posed  in  Chapter  I.  Recommendations  are  made  for 
future  studies. 

B.  CONCLUSIONS 

The  conclusions  of  this  thesis  summarize  the  responses  of  the  four  stakeholder 
groups  -  Tester/Accreditor,  Resource  Sponsor,  Program  Manager,  and  VV&A  Standards 
Official  -  regarding  their  perspectives  on  the  extent  of,  barriers  to,  and  factors  for  PM 
compliance  with  COMOPTEVFORINST  5000.1  A,  and  possible  improvements  that  can 
be  made  to  enhance  the  instruction. 

1.  Extent  of  Compliance  with  COMOPTEVFORINST  5000.1 A 

The  extent  of  compliance  with  COMOPTEVFORINST  5000.1  A  is  most  clearly 
indicated  by  the  stakeholders’  answers  to  the  question:  What  level  of  compliance  with  the 
instruction  do  you  think  was  achieved?4  11  of  14  stakeholders  interviewed  judged 
compliance  to  be  a  “4  out  of  5”  toward  full  compliance.  Thus,  most  of  the  stakeholders 
perceived  a  moderately  high  level  of  compliance  with  the  instruction.  Nevertheless,  a 
noteworthy  minority  saw  some  clear  instances  of  non-compliance  and  provided  evidence 
and  rationale  for  these  observations.  Non-compliance  issues  were  also  noted  within  the 
overall  positive  comments  of  the  majority.  Several  stakeholders  provided  complex 
evaluations,  which  indicated  positives  for  some  areas  of  VV&A  perfonnance  and  pointed 
to  negatives  for  others.  With  this  data,  it  can  be  reasonably  concluded  that  the  level  of 
PM  compliance  with  COMOPTEVFOR’s  VV&A  process  is  good  but  could  be  better. 


4  This  was  asked  in  semi-structured  interview  Question  #2.  Support  for  the  reliability  of  the  appraisals 
provided  by  Question  #2  came  from  the  answers  to  Question  #1,  which  provided  evidence  of  wide-ranging 
stakeholder  VV&A  experiences  and  working  familiarity  with  COMOPTEVFORINST  5000. 1A. 
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2.  Barriers  to  Compliance  with  COMOPTEVFORINST  5000.1A 

The  barriers  to  compliance  with  COMOPTEVFORINST  5000.1  A  are  indicated 
by  the  stakeholders’  answers  to  the  three  interview  questions5  that  dealt  with  the  “costs” 
of  compliance,  the  “risks”  of  compliance,  and  the  “benefits”  of  non-compliance. 

The  first  type  of  barrier  to  full  compliance  with  COMOPTEVFOR’s  VV&A 
process  is  the  perceived  “significant”  costs  borne  by  PMs.  Four  of  the  stakeholders 
interviewed  with  Question  #4  asserted  the  concern  for  costs.  One  minority  comment  in 
particular  pointed  out  that  cost  increases  for  M&S  budgets  are  likely  because  of  the 
“stringent  documentation  and  configuration  management  requirements  demanded  by  the 
instruction.”  Sympathetically,  a  number  of  the  stakeholders  agreed  that  there  were  more 
“upfront”  costs  associated  with  compliance,  particularly  in  the  areas  of  M&S 
development  and  VV&A  activities,  and  especially  for  new  programs.  However,  the 
majority  of  stakeholders  viewed  examples  like  this  to  be  “false”  barriers  because  they 
judged  that  PMs  would  have  to  pay  more  over  the  life  of  their  RDT&E  programs  for 
VV&A  process  non-compliance. 

Risk  was  perceived  to  be  another  (second)  type  of  barrier  to  compliance,  though 
most  stakeholders  who  answered  Question  #5  found  it  to  be  only  a  minor  problem.  But 
one  stakeholder  saw  a  significant  risk  in  complying  with  this  instruction,  should  there  be 
any  last  minute  changes  to  test  requirements/objectives.  This  stakeholder’s  concern  was 
that  such  changes  could  cause  delays  in  accreditation  and  test  schedules,  “because 
stepping  back  would  require  rework  on  many  artifacts,  thus  adding  extra  VV&A 
activities.”  A  few  other  stakeholders  raised  tentative  concerns  about  the  risk  of  investing 
time  and  money  in  conducting  a  diligent  VV&A  effort,  only  to  be  denied  eventual 
approval  because  “the  bar  was  set  to  high.”  As  was  the  case  with  costs,  most 
stakeholders  downplayed  the  barriers  posed  by  perceived  risks  by  pointing  out  that  such 
risks  were  “inherent”  in  any  accreditation  effort  and  would  be  mitigated  by  careful 
planning.  Thus,  most  stakeholders  viewed  the  risk  of  complying  to  be  more  of  an  issue 
of  “perception”  than  a  matter  of  substance. 

5  These  are  Questions  #4,  #5,  and  #6  from  the  semi-structured  interviews. 
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Lastly,  the  interviews  showed  that  a  few  of  the  stakeholders  who  answered 
Question  #6  saw  “benefits”  in  not  complying  with  COMOPTEVFOR  5000.1  A.  These 
stakeholders  believed  that  non-compliance  would  allow  PMs  to  exercise  more  flexibility 
in  their  VV&A  processes.  However,  several  of  the  other  stakeholders  mentioned  that 
“tailoring”  of  the  COMOPTEVFOR  VV&A  process  was  (or  should  be)  a  possibility  for 
PMs  that  would  help  address  the  need  for  flexibility.  However,  the  idea  of  reaping  any 
real  benefit  from  a  significant  non-compliance  with  the  instruction  was  a  “non-starter”  for 
the  overwhelming  majority  of  stakeholders  interviewed. 

3.  Factors  for  Compliance  with  COMOPTEVFORINST  5000.1 A. 

The  factors  for  compliance  with  COMOPTEVFORINST  5000.1  A  are  indicated 
by  the  stakeholders’  answers  to  the  three  interview  questions6  that  dealt  with  the 
“benefits”  of  compliance,  and  the  “costs”  and  “risks”  of  non-compliance. 

In  answering  interview  Question  #3,  the  stakeholders  gave  answers  that  point  to  a 
strong  consensus  that  many  significant  benefits  derived  from  full  PM  compliance  with 
COMOPTEVFORINST  5000.1  A.  The  most  obvious  benefit  of  compliance  cited  was  the 
high  likelihood  of  Tester/Accreditor  (i.e.,  COMOPTEVFOR)  acceptance  of  the  M&S 
proposed  for  use  in  addressing  a  COI  under  OT.  Other  benefits  observed  include  the 
higher  quality  of  the  M&S  being  produced  and  the  increased  potential  to  re-use  the  M&S 
accredited  through  the  process.  One  stakeholder  noted  that  new  participants  who  joined 
later  in  the  VV&A  process  seemed  better  able  to  see  the  rationale  for  the  use  of  the  M&S, 
which  made  for  a  smoother  transition  in  the  program.  Several  of  the  stakeholders 
affirmed  that  the  most  significant  program  benefit  was  the  opportunity  gained  in  defining 
problems  early,  so  that  they  were  addressed  in  time  (and  within  budget).  This  allowed 
PMs  to  either  make  adjustments  within  the  program  or  take  the  program  in  another 
direction  without  major  impact  to  cost  and  schedule.  Many  saw  overall  program  cost 
reduction  and  standardization  of  M&S  products  as  significant  benefits  realized  through 
compliance  with  COMOPTEVFOR’ s  VV&A  instruction. 

6  These  are  Questions  #3,  #7,  and  #8  from  the  semi-structured  interviews. 
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The  answers  to  Questions  #7  and  #8  indicate  that  the  costs  and  risks  entailed  by 
non-compliance  make  strong  dissuaders  for  any  PM  deviations  from  the  instruction. 

The  vast  majority  of  stakeholders  interviewed  considered  the  costs  of  not 
complying  with  the  COMOPTEVFOR  VV&A  instruction  to  be  (at  least)  “several”  and 
“significant.”  In  their  comments,  the  stakeholders  seemed  to  have  no  trouble  coming  up 
with  concrete  (i.e.,  real  dollar  value)  examples  of  the  costs  incurred  by  non-compliance. 
A  major  concern  was  the  cost  fallout  of  the  M&S  not  meeting  COMOPTEVFOR’s 
technical  requirements:  Useful  M&S  would  not  be  produced  at  the  end  of  all  the  PM’s 
expenditures  because  COMOPTEVFOR  found  insufficient  reason  to  be  confident  in  the 
credibility  of  the  M&S.  This  adverse  outcome  could  bring  an  acquisition  program  to  a 
halt,  where  the  cost  to  go  back  and  re-conduct  the  VV&A  will  be  more  expensive  to  the 
U.S.  Navy. 

Similarly,  an  overwhelming  majority  of  the  stakeholders  viewed  the  risks  of  not 
complying  with  the  COMOPTEVFOR  VV&A  instruction  to  be  (at  least)  “several”  and 
“significant.”  A  number  of  “real  life”  examples  were  provided,  which  demonstrated  the 
seriousness  of  the  risks  that  non-compliant  M&S  proponents  would  take  on. 
Stakeholders  noted  that  risks  realized  are  often  “felt”  in  the  form  of  added  costs  and 
extended  schedules.  A  technical  risk  in  observed  for  non-compliance  was  the  creation  of 
“false  positives”  with  regard  to  M&S  credibility.  If  the  PMs  were  to  serve  as  their  own 
accreditors  in  the  VV&A  process,  a  stakeholder  comment  posited,  “they  would  be  less 
stringent  in  meeting  the  M&S  use  requirements  and  would  run  the  risk  of  qualifying  an 
M&S  that  would  be  deficient  (without  realizing  it).”  The  results  of  these  false  positives 
would  be  rolled-up  into  the  overall  system  to  which  the  M&S  is  being  applied,  thus 
embedding  errors  that  could  lead  to  bad  decisions  on  the  operation  of  the  overall  system. 
The  unfortunate  result  is  that  the  warfighter  could  pay  a  high  price  for  those 
miscalculations,  in  some  cases  with  his  or  her  life. 

The  consensus  demonstrated  by  the  stakeholders  on  the  issues  of  cost  and  risk 
may  be  a  good  indication  of  the  effectiveness  of  the  COMOPTEVFOR  instruction  in 
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educating  the  U.S.  Navy  surface  ship  acquisition  participants  on  the  importance  of 
following  the  standards  of  the  VV&A  process. 

4.  Possible  Improvements  to  COMOPTEVFORINST  5000.1A 

Stakeholder  interview  Question  #10  (with  insight  provided  by  Question  #9) 

provides  the  data  for  whether  improvements  are  possible  for  COMOPTEVFOR  5000.1  A. 

The  stakeholders’  comment-answers  indicated  that  no  consensus  existed  regarding  the 

opportunity  for  making  improvements;  rather,  they  indicated  that  there  were  two 

opposing  camps  on  the  issue  (and  possibly  a  third  one  somewhere  in  the  middle).  The 

two  opposing  perspectives  were:  (a)  There  is  no  real  opportunity  (or  impetus)  for 

improvement;  and  (b)  There  is  opportunity  for  making  significant  improvements  to  the 

latest  instruction.  The  first  camp  was  occupied  by  a  minority  (6  of  14)  stakeholders;  the 

second  was  larger  and  more  productive  with  comments.  The  below  comment  from  a 

Tester/Accreditor  pointed  to  an  improvement  that  resonated  among  the  stakeholders: 

Since  OT  is  the  final  step  before  full  production,  it  would  be  most 
efficient  to  offer  the  PM  a  VV&A  instruction  that  contains  both 
policy  and  detailed  “how  to”  guidance  a  single,  usable  resource. 

The  PM  would  there  find  efficient  methods  to  organize,  estimate 
costs,  determine  schedules,  and  finish  a  quality  M&S  on  time  and 
within  budget. 

Thus,  many  of  the  stakeholders  interviewed  appeared  to  be  saying  that 
COMOPTEVFORINST  5000.1  A  was  not  sufficient  unto  itself  for  providing  M&S 
proponents  with  the  information  needed  to  conduct  the  required  VV&A.  Acknowledging 
this  insufficiency,  several  of  the  stakeholders  offered  specific  remedies.  A  few 
respondents  indicated  that  a  “how  to”  guidance  should  be  included  within  the  instruction 
that  would  give  PMs  the  ability  to  establish  a  more  efficient  method  for  organizing, 
estimating  costs,  and  determining  schedule  impacts.  This  would  assist  the  PMs  in 
producing  a  quality  M&S  on  time  and  within  budget.  Another  respondent  suggested  that 
an  appendix  should  be  added  to  the  instruction  that  would  provide  detailed  examples  of 
how  to  conduct  VV&A.  This  appendix  would  give  a  new  PM,  step-by-step  instructions 
and  help  him/her  to  better  understand  the  VV&A  process.  One  other  recommendation 
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was  the  implementation  of  an  on-line  “distributed  knowledge-base”  capability  for  the 
COMOPTEVFORINST  5000.1  A  VV&A  process.  The  respondent  felt  that  this  would 
not  only  be  innovative  to  PMs,  but  would  also  save  money  for  the  VV&A  community  by 
reducing  travel  cost  to  remote  sites  to  attend  meetings. 

C.  RESEARCH  QUESTIONS 

1.  Primary  Research  Question 

The  primary  research  question  posed  by  this  thesis  is:  What  are  the  barriers  to 
compliance  with  COMOPTEVFOR's  VV&A  process  that  U.S.  Navy  surface  ship  (and 
weapons)  acquisition  programs  face? 

A  basic  lack  of  understanding  of  COMOPTEVFOR's  VV&A  process  and  costs 
are  the  primary  barriers  stakeholders  are  facing  with  not  complying  with  this  instruction. 
Some  PMs  are  not  aware  or  lack  the  understanding  that  the  COMOPTEVFORINST 
5000.  lA's  process  calls  for  COMOPTEVFOR  to  get  involve  early  in  their  program's 
acquisition  cycle  to  determine  which  assets  and  resource  are  appropriate  to  support 
operational  testing,  along  with  identifying  the  proper  modeling  and  simulation 
requirements  which  would  include  operational,  programmatic,  and  perfonnance  data. 
Cost  savings  to  programs  would  be  realized  if  COMOPTEVFOR  is  involved  early,  to 
identify  or  resolve  any  issues  that  could  impact  OT. 

Another  barrier  is  "cost"  if  not  adequately  budgeted  by  PMs  for  M&S 
development  and  to  conduct  V&V  activities.  "Upfront"  costs  are  necessary  for  the 
development  of  credible  M&S  which  is  only  obtained  by  developing  a  well-defined 
VV&A  process  for  the  M&S  use.  Investing  early  in  M&S  and  VV&A  efforts  will  result 
in  a  cost  savings  to  the  overall  program. 

2.  First  Subsidiary  Question 

First  subsidiary  question:  What  benefits  and  liabilities  are  seen  by  the 
stakeholders  in  subscribing  to  the  COMOPTEVFORINST  5000.1  A  VV&A  process,  if  its 
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standards  were  consistently  followed  by  all  U.S.  Navy  surface  ship  (and  weapons) 
acquisition  programs? 


a.  Benefits  seen  by  Stakeholders: 

Respondents  said  that  PMs  and  everyone  else  will  gain  a  common 
understanding  of  the  requirements  when  they  consistently  follow  the  process  standards  of 
COMOPTEVORINST  5000.1  A.  This  common  understanding  will  translate  to  more 
efficient  management  of  the  entire  acquisition  process  that  will  endure.  The  sum  result  is 
both  a  larger  dollar  savings  due  to  management  efficiencies  and  a  higher  quality  product 
delivered  in  a  shorter  period  of  time.  In  addition,  compliance  with  the  instruction  would 
help  avoid  delays  of  Defense  Acquisition  Program  Milestone  B  and  the  need  to 
reprogram  funds  for  physical  testing. 

b.  Liabilities  seen  by  Stakeholders: 

One  respondent  perceived  that  if  COMOPTEVFORINST  5000.1  A  was 
consistently  followed,  PMs  would  not  have  the  opportunity  to  research  new 
methodologies  that  could  be  more  beneficial  and  cost-effective  in  supporting  their  OT 
requirement.  Another  respondent  felt  that  PMs  might  see  the  accreditation  process  as  too 
much  work  and  therefore  would  have  no  reason  to  reduce  testing.  This  mindset  would 
provide  leverage  for  the  OTD  to  require  more  testing  to  make  up  for  the  lack  of  support 
that  accredited  M&S  provides.  This  dynamic  saves  money  up  front  but  the  PM  will  pay 
more  later  for  the  additional  testing.  This  is  because  the  PM  will  learn  of  problems  later 
when  they  will  cost  more  to  fix.  Even  worse,  fewer  requirements  will  be  met  because 
some  of  the  problems,  when  realized  for  the  first  time  at  OT,  are  discovered  too  late  to 
correct. 


3.  Second  Subsidiary  Question 

Second  subsidiary  question:  What  are  the  cost  and  schedule  considerations  that 
impact  U.S.  Navy  surface  ship  (and  weapons)  acquisition  programs  in  the  course  of 
complying  with  the  COMOPTEVFORINST  5000. 1 A  VV&A  process? 
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There  are  additional,  upfront  costs  associated  with  M&S  development  and  VV&A 
process  support;  however,  there  are  no  real  added  costs  associated  with  adherence  to  the 
COMOPTEVFOR’s  VV&A  instruction.  In  fact,  most  of  the  stakeholders  interviewed 
argued  that  the  life-cycle  cost  to  use  M&S  for  OT  is  reduced  compared  to  what  would  be 
needed  to  perform  physical  testing  to  prove  COIs.  While  some  PMs  may  view  VV&A  as 
too  formal,  too  cumbersome,  and  even  too  costly  and  time  consuming  for  acquisition 
programs  controlled  by  cost  and  schedule  constraints,  this  view,  as  articulated  by  a 
stakeholder,  appears  to  be  shortsighted  and  unwise  from  the  total  acquisition  cost  and 
schedule  perspective. 

4.  Third  Subsidiary  Question 

Third  subsidiary  question:  What  are  the  risks  observed  by  the  stakeholders  when 
the  COMOPTEVFORINST  5000.1  A  VV&A  process  is  not  followed? 

As  indicated  by  the  stakeholders,  the  risks  associated  with  non-compliance  of  this 
instruction  will  include  missed  opportunities  to  adequately  plan  the  M&S  and  VV&A 
objectives  needed  to  obtain  a  continual  improvement  process  throughout  the  ship’s 
acquisition  life-cycle.  In  addition,  opportunities  would  also  be  lost  in  making  strategic, 
up-front  investments  in  M&S  and  VV&A  such  as  design  &  trade  studies,  pre-test 
planning,  and  pre-test  predictions;  the  lack  thereof  results  in  poor  quality  and 
unaccredited  M&S.  Ultimately,  OT  test  requirements  would  not  be  fully  satisfied  due  to 
a  lack  of  credibility  in  the  M&S. 


5.  Fourth  Subsidiary  Question 

Fourth  subsidiary  question:  What  actions  can  be  taken  by  stakeholders  to  ensure 
compliance  with  COMOPTEVFOR's  VV&A  process? 

In  order  for  the  four  stakeholder  groups  to  ensure  compliance  with 
COMOPTEVFOR's  VV&A  process,  they  must  not  only  do  their  individual  part  but  also 
work  collectively  as  a  group.  For  instance,  the  VV&A  standards  group  should  not  only 
be  responsible  for  the  coordination  and  integration  of  M&S  efforts  across  all  U.S.  Navy 
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warfare  and  support  areas,  but  they  should  also  be  actively  communicating  and 
encouraging  the  use  of  VV&A  standards  to  the  other  stakeholder  groups.  Likewise,  in 
addition  to  identifying  the  operational  requirements  for  the  particular  weapon  system  and 
providing  the  required  funding  to  support  that  system,  the  resource  sponsor  should  be 
routinely  engaged  with  the  PM  to  ensure  that  the  VV&A  requirements  are  being  met  and 
appropriate  funding  is  allocated  in  that  area.  Although  the  PM  is  ultimately  responsible 
for  selecting  or  developing  the  M&S  and  ensuring  that  it  is  accredited  to  support  its 
intended  use,  he/she  should  make  sure  that  COMOPTEVFOR  (tester/accreditor)  is 
involved  in  the  early  stages  of  the  M&S  development  and  VV&A  process  to  provide 
operational  insight  so  that  any  issues  or  constraints  can  be  addressed  and  corrected  early 
enough  to  avoid  any  delays  during  OT.  Finally,  the  tester/accreditor,  whose  main 
concern  is  obtaining  an  M&S  of  sufficiently  high  fidelity  as  it  relates  to  accurately 
modeling  the  physical  behavior  of  the  system  under  test,  should  also  be  sensitive  to  the 
cost  and  schedule  constraints  of  the  acquisition  program.  If  these  synergies  were 
maintained  among  the  stakeholders,  they  will  be  ensuring  compliance  with  the 
instruction. 

D.  RECOMMENDATION 

The  stakeholders,  working  together,  should  diligently  comply  with  the  VV&A 
process  advocated  by  COMOPTEVFORINST  5000.1  A  in  developing  the  M&S  needed  in 
support  of  U.S.  Navy  surface  ship  OT  requirements.  However,  this  can  be  an  issue  for 
PMs  when  the  upfront  costs  to  support  the  requirements  specified  in  the  instruction  are 
not  in  their  original  acquisition  program  budget.  Early  interaction  between  the 
stakeholders,  particularly  between  the  PM  and  the  Tester/Accreditor,  is  crucial  to 
addressing  this  issue.  The  VV&A  Standards  stakeholder  can  provide  considerable 
information  and  support  to  the  PM  and  the  Tester/Accreditor  that  would  help  anticipate 
the  cost  and  schedule  requirements.  Ultimately,  the  Resource  Sponsor  can  broker  the 
expectations  between  the  PM  and  the  Tester/Accreditor  and  approve  the  budget. 

A  solution  that  would  help  absorb  the  possible  VV&A  cost  deltas  above  any  one 
specific  acquisition  program  RDT&E  baseline  would  be  to  establish  a  special  RDT&E 
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funding  line  dedicated  to  the  VV&A  of  M&S  for  OT.  Appropriated  under  OPNAV  N76 
cognizance,  this  RDT&E  funding  would  be  potentially  available  to  all  surface  ship 
acquisition  programs  when  the  need  arises;  the  line  would  be  budgeted  yearly  to  support 
the  PMs  if  the  effort  to  support  the  COMOPTEVFORINST  5000.1  A  VV&A  process  isn’t 
in  their  individual  budgets.  PMs  should  have  to  provide  the  appropriate  justification 
when  requesting  this  funding  to  support  their  OT  requirements.  This  special  RDT&E 
funding  line  would  also  support  M&S  reuse  among  surface  ship  acquisition  programs, 
since  the  N76  Resource  Sponsor  would  likely  encourage  such  efforts. 

Also,  to  better  aid  PMs  and  Resource  Sponsors  in  understanding  the  benefits  and 
advantages  of  complying  with  the  COMOPTEVFORINST  5000.1  A  VV&A  process, 
NSMO,  in  conjunction  with  COMOPTEVFOR,  should  provide  training/seminars  to  the 
surface  ship  PMs  and  OPNAV  on  this  instruction. 

If  these  recommendations  were  to  be  implemented  by  the  U.S.  Navy,  the 
researchers  believe  that  there  would  be  a  higher  incidence  of  full  compliance  by  PMs  in 
utilizing  the  COMOPTEVFORINST  5000.1  A  VV&A  process,  thus  allowing  the  U.S. 
Navy  to  achieve  the  ultimate  goal  of  producing  more  credible  M&S  to  resolve  COIs  that 
satisfy  COMOPTEVFOR’ s  OT  requirements  for  surface  ships  and  their  weapon  systems. 


E.  AREAS  FOR  FURTHER  RESEARCH 

The  following  areas  should  be  investigated  for  potential  benefit  to  the  U.S.  Navy, 
the  other  U.S.  military  services,  and  DoD  components: 

1.  Accrediting  Legacy  M&S 

One  of  the  findings  of  this  thesis  was  why  a  well-defined  VV&A  process  for 
newly  acquired  or  modified  surface  ships  is  important  in  gaining  credibility  for  the  M&S. 
Further  research  in  defining  the  steps  that  should  be  taken  in  collecting  or  re -producing 
the  background  data  that  is  needed  to  validate  legacy  models  could  provide  further 
insight  in  the  importance  of  obtaining  credible  M&S. 
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2.  The  “How  To”  of  VV&A 

This  thesis  focused  on  the  organizational  and  managerial  aspects  of  VV&A  of 
M&S  used  for  OT  of  U.S.  Navy  surface  ships  and  their  weapon  systems.  Work  is  being 
done  by  the  U.S.  Navy’s  VV&A  Standards  group,  namely  NMSO,  to  produce  a  "how  to" 
handbook  on  the  VV&A  process.  The  handbook  therein  is  expected  to  address  a  specific 
need  identified  by  a  number  of  the  stakeholders  interviewed.  Further  research  can  be 
done  by  collecting  data  on  the  VV&A  process  perfonnance  of  stakeholders  before  and 
after  issuance  of  the  Implementation  Handbook  to  further  provide  guidance  to  the 
community  on  M&S  VV&A. 

The  areas  that  are  identified  for  future  research  will  help  to  increase  the 
knowledge-base  of  both  stakeholders  and  members  of  the  M&S  VV&A  community  in 
identifying  new  approaches  and  techniques  that  would  lead  to  more  credible  and  cost- 
effective  M&S  for  not  only  the  U.S.  Navy,  but  other  U.S  Military  services  and  DoD 
components  as  well. 
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APPENDIX:  INTERVIEW  DATA 


A  questionnaire  was  administered  to  14  stakeholders  of  the  U.S.  Navy’s  VV&A 
of  M&S  used  for  surface  ship  OT&E.  It  provided  the  basic  structure  of  the  stakeholder 
interviews,  by  soliciting  answers  -  in  the  form  of  a  score  and  a  comment  -  to  each  of  ten 
questions.  The  interviews  were  conducted  on  a  non- attribution  basis;  the  stakeholders’ 
names  and  U.S.  Navy  commands  are  not  identified  in  order  to  protect  their 
confidentiality.  Instead,  a  letter  designation  (e.g.,  [A]  )  is  assigned  to  each  stakeholder. 
Further,  the  stakeholder  category  (e.g..  Tester  /  Accreditor  {TA}  )  and  management  level 
(e.g.,  Lower  Management  {lm}  )  is  identified  for  each  stakeholder.  Table  A.  1-0,  below, 
lays  out  the  interviewed  stakeholder  identifications  and  the  counts  per  category  and  level. 


Table  A.1-0.  Interviewed  Stakeholder  Identifications  and  Counts 

Stakeholder  Category 

Lower 

Management 

{lm} 

Middle 

Management 

{mm} 

Upper 

Management 

{urn} 

Count  per 

Category 

Tester  /  Accreditor  {TA} 

[A]  [B] 

[C] 

[D]  [E] 

5 

Resource  Sponsor  {RS} 

[H] 

[G] 

[I] 

3 

Program  Manager  {PM} 

[K]  [M] 

[L]  [N] 

[O] 

5 

VV&A  Standards  {VS} 

\P] 

1 

Count  per  Management  Level 

5 

4 

5 

14 

The  questions  asked  and  the  answers  provided  in  the  course  of  the  interviews  are 
recounted  below.  The  first  table  below  each  question  lists  the  choices  of  numerical 
scores  offered  to  the  stakeholders  in  answering  the  question.  The  scores  provided  as 
answers  are  reported  in  the  second  table  below  each  question.  The  stakeholders’ 
comment-answers  for  each  question  are  detailed  in  alphabetical  order  of  stakeholder 
identifier  below  the  second  table  of  each  question.  Question  1  (only)  had  two  sub- 
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questions,  which  were  used  to  elicit  amplifying  information  from  the  stakeholders’  about 
their  specific  experiences  in  VV&A. 


QUESTION  1. 

To  what  extent  are  you  familiar  with  the  VV&A  process  required  by  the  newly  updated 
COMOPTEVFORINST  5000.1  A  of  9  September  2004,  as  it  applies  to  USN  surface  ship 
M&S  used  for  OT&E? 


Table  A.l-1.  Choices  of  Score  Answers  Offered  to  Stakeholders  for  Question  1. 


(5) 

(4) 

(3) 

(2) 

(1) 

very 

moderately 

somewhat 

moderately 

very 

familiar 

familiar 

familiar 

unfamiliar 

unfamiliar 

Table  A.l-2.  Scores  Provided  by  Stakeholders  to  Question  #  1 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

5 

5 

5 

5 

5 

1 

2 

2 

4 

4 

4 

2 

5 

4 

Sub-Questions  for  Question  1: 

a.  What  is  one  particular  case  of  VV&A  of  USN  surface  ship  M&S  used  for 
OT&E  in  which  you  participated  and  what  was  your  role  in  this  case? 

b.  Was  the  superseded  COMOPTEVFORINST  5000.1  (of  5  September 
1995)  in  effect  for  this  case? 

Comments  /  Answers  Provided  by  Stakeholders  for  Question  1 : 

Stakeholder  fAl  {TA,  Im}: 

•  I  am  very  familiar  (at  the  higher  level)  with  the  instruction. 

•  I  participated  in  VV&A  for  Rolling  Airframe  Missile  (RAM).  I  supported 
the  OTD  in  developing  OT  M&S  requirements  and  evaluating  V&V  results.  I  also 
participated  in  VV&A  for  Missile  Defense  (which  had  equipment  test  failures)  and  for 
Tomahawk  (all  sections,  using  five  simulators),  which  enabled  minimal  physical  testing, 
but  required  (all  together)  a  long,  detailed  process. 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  in  effect  for  the  VV&A  efforts  in  which  I  participated.  I  supplemented  the 
instruction  with  our  Surface  Warfare  Guide.  (This  is  what  the  PMs  asked  for  and  needed 
to  accomplish  their  VV&A  responsibilities.) 
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Stakeholder  FBI  {TA,  Im}: 

•  I  am  very  familiar  with  the  instruction. 

•  The  one  particular  case  of  VV&A  in  which  I  participated  was  Tomahawk. 
My  role  in  this  case  was  that  of  Operational  Test  Director  (OTD),  where  I  attended  SCPs 
(vs.  SMBs)  and  was  involved  in  the  review  of  five  PM-produced  VV&A  documents:  (1) 
Accreditation  Plan;  (2)  Accreditation  Report;  (3)  Accreditation  Test  Report;  (4) 
Configuration  Management  Plan;  and  (5)  Overarching  VV&A  Plan.  I  helped  the  PMs 
with  their  VV&A  by  providing  the  “Guide  for  U.S.  Navy  Surface  Ship  VV&A  of  M&S 
for  OT&E.” 

•  The  key  to  successful  M&S  is  user  involvement  in  the  VV&A.  The  user 
of  the  weapons  system  is  the  Fleet  and  COTF  represents  the  user  in  testing  and 
accrediting  their  system. 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  in  effect  for  this  case.  The  last  cycle  of  Tomahawk  testing  /  M&S  was  conducted 
under  the  old  instruction. 

Stakeholder  [Cl  1TA,  mm}; 

•  I  am  very  familiar  with  the  instruction. 

•  I  participated  in  the  VV&A  for  one  ECS  (early  stage)  case,  two  CVN-21 
cases,  one  LPD-17  case,  and  a  number  of  others  that  are  related  to  USN  surface  ship 
programs.  My  role  in  these  cases  was  that  of  COMOPTEVFOR’s  VV&A  manager, 
where  I  reviewed  and  approved  the  VV&A  of  M&S  for  OT&E  documents  submitted  by 
the  USN  Surface  Ship  Program  Managers.  I  am  responsible  for  the  migration  of  the 
COMOPTEVFOR  instruction  that  you  are  studying  (COMOPTEVFORINST  5000.1  A  of 
9  September  2004). 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  in  effect  for  most  of  these  cases,  but  since  I  wrote  the  new  instruction,  it  was  in  effect 
for  the  most  recent  case. 

Stakeholder  FD1  {TA,  um}: 

•  I  am  very  familiar  with  the  instruction. 

•  The  one  particular  case  of  VV&A  in  which  I  participated  was  reviewing 
new  policy  as  it  went  through  the  approval  /  signature  process.  My  role  in  this  case  was 
that  of  newly  appointed,  interim  VV&A  Manager  at  COMOPTEVFOR. 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  not  in  effect  for  this  case.  I  was  a  part  of  approving  the  new  instruction. 
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Stakeholder  [El  {TA,  um}: 

•  I  am  very  familiar  with  the  instruction. 

•  The  one  particular  case  of  VV&A  in  which  I  participated  was  reviewing 
(at  the  senior  management  level)  new  policy  as  it  went  through  the  approval  /  signature 
process.  My  role  in  this  case  was  that  of  developing  the  long-term  strategic  policy  with 
respect  to  the  U.S.  Navy  and  OT&E. 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  not  in  effect  for  this  case.  I  was  a  part  of  approving  the  new  instruction. 

Stakeholder  FG1  IRS,  mm}: 

•  I  am  very  unfamiliar  with  the  instruction. 

•  I  oversaw  the  DD(X)  TEMP  process  from  adjudication  to  signature.  This 
included  general  oversight  of  VV&A  conducted  in  support.  My  responsibilities  include: 

—  Defining  requirements  for  Surface  Navy  (the  DD(X)  program,  in 
particular). 

—  Providing  requirements  to  the  PM  for  product  by  way  of  the 
Capabilities  Design  Document  (CDD). 

—  Reviewing/authorizing  the  PM’s  plan  on  how  the  product  will  be 
implemented. 

—  Providing  funding  to  the  PM  to  implement  the  product. 

—  Maintaining  balance  between  PM  &  COTF  (and  DOT&E)  with  respect 
to  program  costs  &  risks. 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  not  in  effect  for  this  case. 

Stakeholder  [HI  IRS,  Iml: 

•  I  am  moderately  unfamiliar  with  the  instruction. 

•  The  one  particular  case  of  VV&A  in  which  I  participated  was  where  two 
systems  for  Tomahawk  were  evaluated:  the  TTWCS  upgrade  ver  5  and  the  TC2S  ver  4.1. 
My  role  in  this  case  was  that  of  Resource  Sponsor  for  Tomahawk  Weapons  System. 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  not  in  effect  for  this  case. 

Stakeholder  [II  IRS,  um}: 

•  I  am  moderately  unfamiliar  with  the  instruction. 

•  There  are  a  number  of  cases  of  VV&A  in  which  I  participated: 
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—  USS  Virginia  (SSN  774)  Program  VV&A  of  a  Transient  Shock 
Analysis  (TSA)  “Process.”  (This  was  recently  completed  and  allowed 
elimination  of  a  $70  Million  dollar  Full  Ship  Shock  Trial.) 

--  DD(X)  ASCM  PRA  Test  Bed.  (This  is  ongoing.) 

—  Joint  Lightweight  Standoff  Chemical  Agent  Detector  (JLSCAD). 

—  Netcentric  Warfare  Knowledge  Sharing. 

-  LPD-17,  LHA-6.  and  CVN-21 

•  My  role  in  these  cases  was  that  of  the  U.S.  Navy’s  T&E  oversight,  where  I 
review  the  VV&A  to  see  that  the  M&S  is  used  appropriately  for  T&E  (whether  for 
DT&E  or  OT&E).  While  COTF  is  a  tester,  OPNAV  N091  is  not  (though  we  were). 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  in  effect  for  the  older  cases;  but  was  not  was  not  in  effect  for  the  most  recent  cases. 

Stakeholder  [K1  1PM,  Im}: 

•  I  am  moderately  familiar  with  the  instruction. 

•  The  one  particular  case  of  VV&A  in  which  I  am  participating  is  the 
DD(X)  program,  though  it  is  not  at  that  stage  yet.  However,  we  are  planning  for  the 
VV&A  to  the  extent  possible.  My  role  in  this  case  was  that  of  project  engineer. 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995)  is 
not  in  effect  for  this  case.  Thus,  we  are  using  the  new  instruction. 

Stakeholder  [LI  1PM,  mm}: 

•  I  am  moderately  familiar  with  the  instruction. 

•  The  one  particular  case  of  VV&A  in  which  I  participated  was  the  Threat  D 
M&S  Suite  to  support  TEMP  124/801  OT-IIG.  My  role  in  this  case  was  that  of 
managing  the  VV&A  of  the  Threat  D  M&S  Suite  to  support  TEMP  124/801  OT-IIG. 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  not  in  effect  for  this  case. 

Stakeholder  [Ml  1PM,  Im}: 

•  I  am  moderately  familiar  with  the  instruction. 

•  The  one  particular  case  of  VV&A  in  which  I  participated  was  for  the 
LPD-17  Probability  of  Raid  Annihilation  (PRA)  V&V.  My  role  in  this  case  was  that  of 
program  analyst. 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  in  effect  for  this  case. 
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Stakeholder  [N1  {PM,  mm}; 

•  I  am  moderately  unfamiliar  with  the  instruction. 

•  The  one  particular  case  of  VV&A  in  which  I  participated  was  LPD-17 
Probability  of  Raid  Annihilation  (PRA)  for  ship  self  defense.  My  role  in  this  case  was 
that  of  supporting  PM  in  the  development  of  the  M&S  and  contributing  to  the  VV&A 
case. 


•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  in  effect  for  this  case. 

Stakeholder  FOl  {PM,  uml: 

•  I  am  very  familiar  with  the  instruction. 

•  The  one  particular  case  of  VV&A  in  which  I  participated  was 
T echEval/ OpEval  for  CEC;  OT  for  ESSM;  OT  for  SPYID(V);  and  ABMD  for  COTF. 
My  role  in  this  case  was  that  of  overseeing  the  VV&A  processes  and  assisting  the  PMs  in 
performing  VV&A. 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  in  effect  for  these  cases. 

Stakeholder  [PI  {VS,  um}; 

•  I  am  moderately  familiar  with  the  instruction. 

•  I  participated  in  all  of  the  recent  VV&A  cases,  as  I  provide  the  Navy’s 
oversight  on  all  VV&A  and  communicate  /  coordinate  the  Navy’s  standards  on  VV&A. 

•  The  superseded  COMOPTEVFORINST  5000.1  (of  5  September  1995) 
was  not  in  effect  for  the  most  recent  cases. 

QUESTION  2. 

What  level  of  compliance  with  the  (effective)  instruction  do  you  think  was  achieved  in 
this  case? 

Comments  Provided  by  Stakeholders  for  Question  2: 

Stakeholder  fAl  {TA,  Im}; 

•  I  think  that  1 00%-9 1  %  compliance  was  achieved. 

•  A  higher  level  of  compliance  was  achieved  for  those  PMs  who  used 
COTF’s  supplemental  Surface  Warfare  VV&A  Guide. 

•  The  process  diagram  developed  by  NDAT  was  helpful  in  establishing  a 
standard  process  and  schedule. 
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Table  A.2-1.  Choices  of  Score  Answers  Offered  to  Stakeholders  for  Question  2. 


(5) 

(4) 

(3) 

(2) 

(1) 

100-91% 

90-81% 

80-71% 

70-61% 

60%  or  less 

compliance 

compliance 

compliance 

compliance 

compliance 

Table  A.2-2.  Scores  Provided  by  Stakeholders  to  Question  #  2 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

5 

5 

4 

1 

1 

1 

5 

5 

4 

5 

4 

4 

5 

5 

Stakeholder  FBI  {TA,  lm}: 

•  I  think  that  1 00%-9 1  %  compliance  was  achieved. 

•  The  VV&A  efforts  of  one  particularly  successful  program  consistently 
demonstrated  a  high  level  of  compliance  with  respect  to  “content”,  though  not  as  high 
with  respect  to  “schedule.”  The  real  objective  for  compliance  is  to  have  “no  surprises” 
with  respect  to  the  content  of  the  VV&A  produced  for  the  M&S. 

•  Successful  compliance  was  achieved  when  a  COTF  representative 
participated  on  the  SCP. 

•  Both  PMs  and  COTF  were  usually  not  successful  in  meeting  VV&A 
timelines.  Then  again,  failing  to  meet  the  original  schedule  is  usually  not  the  real 
problem. 

Stakeholder  [Cl  {TA,  mm}: 

•  I  think  that  90%-8 1  %  compliance  was  achieved. 

•  As  the  new  instruction  has  hardly  been  in  place,  no  one  has  yet  had  the 
opportunity  to  demonstrate  a  level  of  compliance  for  it. 

Stakeholder  FD1  IT  A,  um}: 

•  I  think  that  60%  or  less  compliance  was  achieved. 

•  Meeting  VV&A  schedules  is  an  issue. 

•  In  one  case,  the  PM  did  not  provide  the  data  to  finish  the  VV&A  package. 
So  the  data  had  to  be  developed  by  the  OT  himself. 

•  The  old  instruction  (which  applied)  was  not  very  effective.  The  new 
instruction  will  be  more  effective. 
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Stakeholder  [El  {TA,  um}: 

•  I  think  that  60%  or  less  compliance  was  achieved. 

•  A  very  low  level  of  compliance  was  achieved.  Most  PMs  did  not  seem  to 
be  knowledgeable  about  the  instruction.  Timing  (i.e.,  late  delivery)  was  the  usual  cause 
of  non-compliance. 

Stakeholder  FG1  IRS,  mm}: 

•  I  think  that  60%  or  less  compliance  was  achieved. 

•  A  case  of  low  compliance  was  recognized  when  the  Resource  Sponsor 
(RS)  ran  into  a  big  issue  at  the  end  of  the  VV&A  process:  DOT&E  was  not  satisfied  with 
the  capability  of  the  M&S  to  credibly  predict  the  perfonnance  of  the  weapon  system 
modeled.  The  RS  wanted  to  use  M&S  to  avoid  the  cost  and  risk  entailed  in  conducting  a 
certain  close-in  live-fire  testing  event.  But  the  PM  had  not  engaged  the  TA  sufficiently 
(or  early  enough)  in  their  VV&A  process  to  identify  the  shortcomings  of  the  M&S,  such 
that  DOT&E  rejected  the  use  of  the  M&S  for  the  close-in  event.  Consequently,  the  RS 
had  to  reprogram  $250M  to  conduct  a  physical  test  that  the  M&S  was  hoping  to  avoid. 

Stakeholder  [HI  IRS,  Im}: 

•  I  think  that  1 00%-9 1  %  compliance  was  achieved. 

•  The  maturity  and  experience  of  the  team  that  develops  the  ship’s  T&E 
program,  which  includes  M&S  in  its  mix,  has  a  lot  to  do  with  any  successful  compliance 
with  the  VV&A  instruction. 

Stakeholder  [II  IRS,  um}: 

•  I  think  that  1 00%-9 1  %  compliance  was  achieved. 

•  One  recently  completed  VV&A  effort  enabled  the  elimination  of  a  $70M 
physical  T&E  event  required  for  a  major  ship  acquisition.  This  success  was  achieved 
because  the  PM  employed  a  “test-model-test”  iterative  effort  over  a  number  of  years  in 
the  ship’s  RDT&E  program  to  collectively  understand  the  physics  of  physical  event  on 
the  ship.  Further,  the  ship’s  designs,  tests,  and  models  were  planned  to  work  in  support 
of  each  other.  The  ship’s  subassembly  construction  techniques  facilitated  M&S  and 
VV&A.  Finally,  M&S  process  control  was  achieved  to  at  least  CMMI  level  3  standards. 
This  facilitated  a  rapid  VV&A  process. 

Stakeholder  [K1  1PM,  Im}: 

•  I  think  that  at  least  90-81%  compliance  will  be  achieved. 

•  The  PM  is  trying  to  achieve  a  high  level  of  compliance  (at  least  90% 
compliance)  and  -  among  other  efforts  -  attended  VV&A  TWG  meetings  to  learn  how  to 
do  so. 
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Stakeholder  [LI  {PM,  mm}: 

•  I  think  that  1 00%-9 1  %  compliance  was  achieved. 

•  The  PM  worked  very  closely  with  the  OTD  to  ensure  that  the  PM’s 
process  was  in  accordance  with  COTF  policy  and  met  their  requirements. 

Stakeholder  [Ml  1PM,  Im}: 

•  I  think  that  at  least  90-81%  compliance  will  be  achieved. 

•  The  COTF  instruction  is  fairly  general.  The  PM  believes  that  his  VV&A 
program  will  meet  the  instruction’s  intent.  COTF  was  engaged  early  on  in  providing 
input  to  the  process.  However,  the  instruction  was  not  written  with  the  scale  of  the 
program’s  M&S  in  mind. 

Stakeholder  [N1  1PM,  mm}: 

•  I  think  that  at  least  90-81%  compliance  will  be  achieved. 

•  The  VV&A  case  on  which  the  PM  is  working  is  still  under  development. 
The  PM  thinks  that  he  will  be  highly  compliant. 


Stakeholder  FOl  1PM,  uml: 

•  I  think  that  1 00%-9 1  %  compliance  was  achieved. 

•  The  PM  believes  that  he  achieved  a  high  level  of  compliance  for  the 
VV&A  efforts  in  which  he  participated. 

Stakeholder  [PI  {VS,  um}: 

•  I  think  that  1 00%-9 1  %  compliance  was  achieved. 

•  It  appears  that  a  high  level  of  compliance  is  being  achieved,  but  it  can  be 
done  more  consistently  and  efficiently. 

QUESTION  3. 

What  do  you  see  to  be  the  benefits  associated  with  compliance  with  this  instruction? 
Comments  Provided  by  Stakeholders  for  Question  3: 

Stakeholder  fAl  {TA,  lm}: 

•  The  benefits  of  complying  with  the  COTF  instruction  are  augmented  when 
PMs  use  the  M&S  Surface  Warfare  Guide  supplemented  by  the  COTF  OTDs.  The  guide 
helps  PMs  (and  everyone  else)  better  understand  what  is  required  (especially  any  OT 
related  clarifications  /  details).  The  combined  use  of  the  instruction  and  the  guide  enables 
more  efficient  management  of  the  entire  process  over  the  years.  Consequently,  there  are 
significant  financial  savings  due  to  the  management  efficiencies.  Also,  a  higher  quality 
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product  is  delivered  in  a  shorter  period  of  time,  which  again  provides  a  financial  savings 
(that  is  obvious  but  hard  to  quantify). 


Table  A.3-1.  Choices  of  Score  Answers  Offered  to  Stakeholders  for  Question  3. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a  few 

no 

significant 

significant 

significant 

minor 

real 

benefits 

benefits 

benefit 

benefits 

benefit 

Table  A.3-2.  Scores  Provided  by  Stakeholders  to  Question  #  3 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

5 

5 

5 

4 

5 

4 

4 

5 

3 

5 

4 

5 

5 

4 

Stakeholder  FBI  {TA,  Im{; 

•  Benefits  seen  from  compliance  include: 

—  A  higher  quality  of  M&S  product  is  obtained. 

—  A  better  assurance  that  M&S  will  meet  the  needs  of  the  user. 

—  OTDs  are  satisfied  that  OT  requirements  are  being  met. 

Stakeholder  [Cl  {TA,  mm}: 

•  Re-use  of  the  M&S  is  one  of  the  many  benefits.  When  the  PMs  comply 
with  the  effective  VV&A  process,  COTF  will  eventually  certify  the  use  of  the  M&S.  The 
accreditation  process  is  built  in.  When  the  program  complies  with  the  instruction,  any 
new  participants  who  join  later  in  the  process  can  readily  see  the  rationale  for  the  use  of 
the  M&S. 

Stakeholder  [D1  {TA,  urn}: 

•  Defining  the  intended  use  of  the  model  is  the  most  important  benefit. 
Compliance  with  the  instruction  gives  the  proponent  a  mechanism  to  support  the 
appropriate  use  of  the  M&S  and  bring  the  process  to  a  recognized  conclusion. 

Stakeholder  [El  {TA,  urn): 

•  Compliance  leads  to  obtaining  more  validated  data  that  other  proponents 
can  use  (i.e.,  re-use). 

•  The  bottom-line  is  that  complying  with  the  instruction  successfully  obtains 
the  accreditor’s  acceptance  of  the  use  of  the  M&S. 


126 


Stakeholder  TGI  {RS,  mm}: 

•  Compliance  with  the  instruction  helps  PMs  avoid: 

—  DODI  5000.2  [particularly  sections  E5.1  and  E5.10]. 

—  Delays  in  reaching  Defense  Acquisition  Milestone  B. 

—  The  need  to  reprogram  RDT&E  funds  to  support  physical  testing 
(vs.  M&S). 

Stakeholder  [HI  {RS,  lmj: 

•  Compliance  helps  promote  standardization  and  commonality.  When  you 
standardize,  you  save  costs  (and  cost  reduction  is  one  benefit). 

•  One  particular  Navy  weapons  program  made  heavy  use  of  M&S  for  tens 
of  years.  What  is  in  place  today  has  saved  a  lot  of  money.  The  program  has  end-to-end 
testing  and  M&S  capability,  which  provides  flexibility  in  using  different  modes,  and  uses 
an  iterative  model  update  process.  The  program  is  able  to  work  a  combination  of 
modeling  and  simulation,  where  the  simulation  generates  environmental  inputs  into  real 
hardware  so  that  the  weapon’s  mission  can  be  run  in  a  very  realistic  way. 

•  One  of  the  significant  program  benefits  of  full  compliance  is  that  problems 
are  defined  early,  so  that  they  can  be  addressed  in  time  and  within  budget. 

Stakeholder  [II  IRS,  urn}: 


•  Compliance  with  the  COTF  instruction  supports  compliance  with  several 
other  DoD  and  DoN  regulations  for  Defense  Systems  Acquisition  that  address  T&E  and 
M&S,  including: 

—  DODI  5000.2  [particularly  sections  E5.1  and  E5.10]. 

--  SECNAVINST  5000. 2C  [particularly  section  5. 4. 7. 9]. 

-  SECNAVINST  5200.40. 

Stakeholder  [K1  {PM,  lm}; 

•  Compliance  entails  earlier  involvement  in  the  definition  of  high-level 
M&S  requirements  and  their  associated  acceptance  criteria.  This  obtains  considerable 
time  and  cost  savings  in  the  long  run. 

Stakeholder  [LI  1PM,  mm}: 

•  COTF  required  a  high  degree  of  rigor  in  their  VV&A  process  (as  called 
out  in  their  instruction)  and  were  supportive  and  helpful  to  the  PM  in  achieving  that  rigor. 
As  a  consequence,  they  “forced”  the  PM  to  be  sure  that  his  data  collection  methodology, 
analysis  processes,  and  reporting  procedures  were  sufficient  to  meet  the  demands  of  the 
Accreditation  Review  Panel  (ARP).  Consequently,  the  PM  is  100%  certain  that  his 
VV&A  would  stand  up  to  outside  audit. 
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Stakeholder  TM1  {PM,  lm}: 

•  By  complying  with  the  instruction,  the  PM  interacted  with  COTF,  the 
accreditation  authority,  and  benefited  from  their  experience  and  guidance. 

Stakeholder  INI  {PM,  mm}: 

•  Compliance  with  the  instruction  promotes  standardization  and  reuse, 
where  it  would  otherwise  only  be  done  at  a  low  percentage.  This  benefits  more  than  the 
individual  program  and  sets  the  groundwork  for  others  to  build  and  reuse. 

Stakeholder  jOl  {PM,  urn}: 

•  There  are  many  benefits  to  compliance,  including  that  it  allows  re-use  of 
M&S  and  it  promotes  proper  documentation. 

Stakeholder  [PI  {VS,  urn}: 

•  The  benefits  of  compliance  come  under  the  categories  of  (1)  Cost- 
reduction  and  (2)  Standardization. 


QUESTION  4. 

What  do  you  see  to  be  the  costs  associated  with  compliance  with  this  instruction? 


Table  A.4-1.  Choices  of  Score  Answers  Offered  to  Stakeholders  for  Question  4. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a  few 

no 

significant 

significant 

significant 

minor 

real 

costs 

costs 

cost 

costs 

cost 

Table  A.4-2.  Scores  Provided  by  Stakeholders  to  Question  #  4 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

4 

4 

1 

2 

2 

2 

2 

1 

2 

3 

1 

4 

4 

1 

Comments  Provided  by  Stakeholders  for  Question  4: 

Stakeholder  [A]  {TA,  lm}: 

•  I  see  several  significant  costs  associated  with  compliance. 

•  When  it  comes  to  compliance,  there  is  no  “free  lunch.”  In  conducting 
VV&A,  due  diligence  (in  planning,  organizing,  executing,  and  documentation)  costs 
money. 
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Stakeholder  FBI : 

•  I  see  several  significant  costs  associated  with  compliance. 

•  Costs  are  incurred  in  both  the  financial  and  schedule  realms,  and  are 
particularly  evident: 

—  During  development  of  the  M&S. 

—  When  performing  the  V&V. 

—  During  collection  and  review  of  data. 

—  At  feedback  to  stakeholders. 

—  With  the  group  doing  the  M&S  when  it  is  not  meeting  the  overall 
schedule. 

•  New  models  require  paradigm  shifts  (incurring  costs  and  schedule 
increases).  Legacy  models  are  more  readily  accepted  and  take  less  time  to  be  accepted. 

Stakeholder  [Cl: 

•  I  see  no  real  cost  associated  with  compliance. 

•  PMs  see  significant  costs  from  compliance  in  having  to  expend  ‘political 
capital’  and  having  to  pay  more  for  “internal  education  processes.” 

•  There  are  costs  in  preparing  the  VV&A  documentation.  Compliance 
requires  a  high  level  of  interaction  between  COTF  and  the  PM  (entailing  planning  costs) 
that  achieves  early,  up-front  recognition  of  the  entire  life-cycle  cost  of  the  M&S.  This  is 
needed,  as  it  should  avoid  the  cost  surprises  often  discovered  down  the  OT&E  road. 

•  Ultimately,  there  are  no  genuine  added  costs  associated  with  compliance. 
It  could  be  argued,  further,  that  the  total  cost  to  use  M&S  for  OT&E  is  probably  less  than 
that  which  would  be  needed  to  perform  physical  testing. 

Stakeholder  FD1: 

•  I  see  a  few  minor  costs  associated  with  compliance. 

•  Any  costs  to  comply  are  minor  because  the  PM  should  otherwise  be  doing 
(e.g.,  for  his  program’s  DT&E)  what  the  COTF  instruction  requires  for  OT&E.  Thus,  the 
instruction  provides  a  framework  for  doing  VV&A  correctly.  Compliance  with  the 
COTF  instruction  supports  a  methodology  that  is  consistent  with  all  acceptable  W&A 
efforts. 

Stakeholder  [El: 

•  I  see  a  few  minor  costs  associated  with  compliance. 

•  The  PM  would  end  up  paying  about  the  same  costs,  regardless  of  whether 
they  are  trying  to  comply  with  the  COTF  instruction,  in  any  genuine  effort  to  prove  that 
their  models  are  credible  for  the  intended  use. 
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•  The  question  to  be  answered  when  deciding  whether  or  not  to  use  an  M&S 
for  OT&E  is:  “What  money  would  be  saved  in  OT&E  if  this  M&S  were  to  be  used 
instead  of  a  physical  test?” 

Stakeholder  [G1 : 

•  I  see  a  few  minor  costs  associated  with  compliance. 

•  Compliance  with  the  instruction  means  more  “up-front  time”  and  man¬ 
hours  devoted  to  PM  -  COTF  interactions  than  would  the  typical  PM  (in  all  likelihood) 
devote  to  VV&A  conducted  without  the  obligation  of  the  instruction.  However,  without 
the  instruction,  the  PM  would  end  up  having  to  collect  more  ‘real’  data  (thereby  incurring 
more  cost)  to  make  the  M&S  credible. 

Stakeholder  [HI: 

•  I  see  a  few  minor  costs  associated  with  compliance. 

•  Compliance  may  bring  about  higher  up-front  costs,  but  down-stream  costs 
(and  consequently  total  costs)  are  lower. 

•  Mature  (seasoned)  acquisition  programs  meet  routinely  with  COTF. 
Consequently,  their  program  costs  for  VV&A  are  lower  as  these  meetings  are  simply  a 
part  of  their  program.  But  for  new  programs,  the  VV&A  meeting  costs  would  likely  be 
higher. 

Stakeholder  [II: 

•  I  see  no  real  cost  associated  with  compliance. 

•  The  cost  of  compliance  is  not  the  issue.  Rather,  the  focus  should  be  on 
meeting  the  technical  challenges. 

•  While  some  PMs  may  view  VV&A  as  too  fonnal,  too  cumbersome,  too 
costly  and  time  consuming  to  a  program  controlled  by  cost  and  schedule  constraints,  this 
view  is  shortsighted  and  not  true  from  the  total  acquisition  cost  and  schedule  perspective. 

Stakeholder  [Kl: 

•  I  see  a  few  minor  costs  associated  with  compliance. 

•  The  stringent  documentation  and  configuration  management  requirements 
demanded  by  the  instruction  will  most  likely  increase  costs  for  the  developers. 

Stakeholder  [LI: 

•  I  see  significant  cost  associated  with  compliance. 

•  Doing  VV&A  right  took  time.  From  start  to  finish  it  took  over  thirteen 
months  to  move  a  very  complex  M&S  through  the  hurdles.  In  this  case,  a  mix  of 
government  and  contractor  personnel  already  under  task  was  used  to  perform  virtually  all 
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phases  of  the  VV&A.  Hiring  outside  (new)  support  would  have  significantly  increased 
the  financial  cost. 

•  It  appears  that  some  DoN  field  activities  are  making  VV&A  unnecessarily 
complex  in  order  to  create  their  own  cottage  industry  of  VV&A  experts.  (This  creates 
unnecessary  costs.) 

Stakeholder  [Ml; 

•  I  see  no  real  cost  associated  with  compliance. 

•  The  instruction  is  so  general  that  any  good  engineering  practice  would 
result  in  compliance.  (Thus,  compliance  entails  no  additional  costs.) 

Stakeholder  FN1: 

•  I  see  several  significant  costs  associated  with  compliance. 

•  There  were  significant  costs  for  one  program,  but  costs  further  down  the 
line  will  be  reduced  because  of  the  introduction  of  an  enterprise  solution  for  multiple 
programs.  The  enterprise  VV&A  plan  will  enable  this  to  become  a  cost  savings  reality. 

Stakeholder  FOl : 

•  I  see  several  significant  costs  associated  with  compliance. 

•  The  costs  associated  with  compliance  depend  on  risk  and  the  limitations  of 
the  M&S. 

Stakeholder  [PI: 

•  I  see  no  real  cost  associated  with  compliance. 

•  Compliance  with  the  instruction  really  provides  a  savings  to  the  DoN. 

QUESTION  5. 

What  do  you  see  to  be  the  risks  associated  with  compliance  with  this  instruction? 


Table  A.5-1.  Choices  of  Score  Answers  Offered  to  Stakeholders  for  Question  5. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a  few 

no 

significant 

significant 

significant 

minor 

real 

risks 

risks 

risk 

risks 

risk 

Table  A.5-2.  Scores  Provided  by  Stakeholders  to  Question  #  5 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

3 

1 

2 

1 

2 

2 

2 

1 

3 

1 

1 

2 

2 

1 
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Comments  Provided  by  Stakeholders  for  Question  5: 

Stakeholder  [A1  {TA,Im}: 

•  I  see  one  significant  risk  associated  with  compliance. 

•  One  significant  risk  in  complying  with  the  instruction  is  that  PMs 
sometimes  see  M&S  accreditation  as  a  reason  to  reduce  future  testing.  With  less  testing, 
there  is  less  new  data  to  plug  into  future  M&S.  Consequently,  the  programmers  fall  into 
the  trap  of  “fixing”  the  next  M&S  to  correlate  with  the  previous,  limited  (and  possibly 
less  relevant)  test  data  set.  The  margin  of  error  between  the  results  of  a  test  and  the 
output  of  its  M&S  is  likely  to  widen,  meaning  that  the  M&S  has  become  of  lower  quality. 
This  can  lead  to  significantly  higher  downstream  costs  due  to  all  the  problems  associated 
with  poor  quality. 

Stakeholder  FBI : 

•  I  see  no  real  risk  associated  with  compliance. 

Stakeholder  [Cl: 

•  I  see  a  few  minor  risks  associated  with  compliance. 

•  There  are  political  risks,  which  are  inherent  in  the  business  of  VV&A. 
Compliance  with  the  instruction  makes  some  of  the  big  risks  go  away,  but  at  the  same 
time  introduces  a  few  new,  smaller  ones. 

Stakeholder  FD1: 

•  I  see  no  real  risk  associated  with  compliance. 

•  The  greatest  risk  the  PM  could  encounter  by  dutifully  complying  with  the 
instruction  would  be  to  discover,  through  the  M&S  for  OT&E  process,  that  the  system  to 
be  acquired  does  not  deliver  the  required  capability  and  consequently  should  not  be 
fielded.  But  this  is  precisely  what  OT&E  is  intended  to  discover  and  thus  an  inherent  risk 
(whether  M&S  or  physical  testing  is  used)  for  the  PM  who  presents  his  system  for 
OT&E. 

Stakeholder  [El: 

•  I  see  no  real  risk  associated  with  compliance. 

Stakeholder  [G1 : 

•  I  see  a  few  minor  risks  associated  with  compliance. 

•  Early  interaction  with  the  Tester  /  Accreditor  could  require  more  work  by 
the  PM  than  he  planned  and  could  drive  some  of  his  M&S  costs. 

•  Resource  sponsors  do  not  engage  at  the  level  of  the  instruction  to 
determine  compliance  and  non-compliance  of  VV&A  requirements.  Rather,  they  seek 
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early  interaction  with  the  PM  and  the  Tester  /  Accreditor  to  determine  the  front  lines  of 
risk  and  cost.  If  the  M&S  issues  do  not  get  figured  out  by  Milestone  B,  then  the  Tester  / 
Accreditor  can  stop  the  process.  The  risk  from  non-compliant  VV&A  is  that  the  use  of 
M&S  could  be  disapproved  late  in  the  program,  which  would  cause  significant  cost 
increases  and  schedule  delays. 

Stakeholder  [HI: 

•  I  see  a  few  minor  risks  associated  with  compliance. 

•  With  mature  programs,  there  are  no  major  risks.  But  at  the  PM  or  ‘trench’ 
level,  there  can  be  concerns  about  the  programmatic  impacts  of  the  ‘outside’  evaluation 
required  by  the  instruction. 

Stakeholder  [II: 

•  I  see  no  real  risk  associated  with  compliance. 

•  As  with  the  cost  question,  the  risk  of  compliance  is  not  the  issue.  Rather, 
the  focus  should  be  on  the  technical  challenges. 

•  The  PM  often  views  M&S  (and  consequently  its  VV&A)  as  merely  a  tool 
to  meet  a  requirement;  rather,  it  is  an  integral  process  within  acquisition  that  reduces 
program  risk. 

Stakeholder  [Kl: 

•  I  see  one  significant  risk  associated  with  compliance. 

•  There  is  the  risk  of  last  minute  changes  to  test  requirements  or  objectives, 
which  may  cause  delays  in  accreditation  (and  thus  test  schedule)  because  stepping  back 
the  process  will  require  rework  on  many  artifacts  and  add  extra  V&V  activities. 

Stakeholder  [LI: 

•  I  see  no  real  risk  associated  with  compliance. 

•  There  is  more  risk  in  not  complying  with  the  COTF  direction  than  there  is 
in  the  act  of  compliance.  If  the  PM  doesn’t  comply,  then  his  VV&A,  and  hence  his 
M&S,  will  not  be  valid  to  use  for  OT.  What  could  be  more  risky  than  developing  M&S 
that  has  no  use  in  OT? 

Stakeholder  [Ml; 

•  I  see  no  real  risk  associated  with  compliance. 

•  Meeting  the  intent  of  the  instruction  is  concomitant  with  good  engineering 
practice.  (Thus,  risk  is  appropriately  managed.) 
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Stakeholder  [Nl: 

•  I  see  a  few  minor  risks  associated  with  compliance. 

•  VV&A  planning  should  go  hand  in  hand  with  engineering  software 
development.  No  significant  risk  should  arise  with  compliance.  However,  there  are 
minor  risks  that  come  from  the  instruction’s  effect  of  driving  PMs  to  comply  to  processes 
that  produce  relatively  ‘safe’  (less  pioneering)  outcomes  rather  than  encouraging  these 
PMs  to  tackle  difficult  technical  issues  that,  while  they  may  not  lead  to  as  high  a 
percentage  of  compliance,  still  produce  greater  M&S  capability. 

Stakeholder  FOl : 

•  I  see  a  few  minor  risks  associated  with  compliance. 

•  If  PMs  comply,  then  the  risks  for  their  VV&A  are  very  limited. 

Stakeholder  [PI: 

•  I  see  no  real  risk  associated  with  compliance. 

•  There  is  no  real  risk  associated  with  compliance  because  all  OT&E 
programs  should  be  in  compliance. 


QUESTION  6. 

What  do  you  see  to  be  the  benefits  associated  with  non-compliance  with  this  instruction? 


Table  A.6-1.  Choices  of  Score  Answers  Offered  to  Stakeholders  for  Question  6. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a  few 

no 

significant 

significant 

significant 

minor 

real 

benefits 

benefits 

benefit 

benefits 

benefit 

Table  A.6-2.  Scores  Provided  by  Stakeholders  to  Question  #  6 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

3 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2 

1 

1 

1 

Comments  Provided  by  Stakeholders  for  Question  6: 

Stakeholder  fAl  1TA,  Im}: 

•  I  see  one  significant  benefit  associated  with  non-compliance. 

•  From  the  OT  perspective,  there  may  be  (perversely)  one  significant  benefit 
associated  with  non-compliance.  In  the  event  the  PM  has  not  complied,  he  has  thus  not 
justified  his  proposal  to  reduce  the  physical  testing  required  for  OT.  This  provides 
leverage  for  the  OTD  to  require  more  testing  to  make  up  for  the  lack  of  support  that 
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accredited  M&S  provides.  So,  while  non-compliance  may  save  money  on  M&S  at  the 
front  end,  the  PM  ends  up  paying  more  for  testing  at  the  back  end.  Moreover,  physical 
testing  uncovers  problems  later  when  they  cost  more  to  fix  (if  they  can  be  fixed  at  that 
point),  which  risks  meeting  fewer  requirements. 

Stakeholder  [B1 ; 

•  I  see  no  real  benefit  associated  with  non-compliance. 

Stakeholder  [Cl: 

•  I  see  no  real  benefit  associated  with  non-compliance. 

•  Non-compliance  produces  no  real  benefit.  Rather,  the  VV&A  participants 
should  have  a  fonnal  understanding  of  the  model’s  intended  use.  They  should  identify 
the  ‘use  cases’.  It  is  important  to  understand  the  implications  of  using  one  type  of  model 
over  another.  The  behavior  of  the  model  should  be  consistent  with  our  understanding  of 
the  physical  phenomena.  The  Configuration  Manager  has  to  be  clear  on  intended  use. 

Stakeholder  FD1: 

•  I  see  no  real  benefit  associated  with  non-compliance. 

Stakeholder  [El: 

•  I  see  no  real  benefit  associated  with  non-compliance. 

Stakeholder  FG1: 

•  I  see  no  real  benefit  associated  with  non-compliance. 

•  There  is  no  real  benefit  that  would  come  from  avoiding  early  interaction 
with  COTF  to  help  make  the  M&S  credible. 

Stakeholder  [HI: 

•  I  see  no  real  benefit  associated  with  non-compliance. 

•  In  practice,  there  isn’t  any  real  benefit  from  non-compliance. 

Stakeholder  [II: 

•  I  see  no  real  benefit  associated  with  non-compliance. 

•  This  question  doesn’t  really  make  sense,  because  non-compliance  with 
instruction  brings  no  benefit  whatsoever. 

•  Unfortunately,  PMs  at  the  start  of  their  programs  often  don’t  know  what 
they  want  to  use  M&S  for.  Full  compliance  is  difficult  when  the  need  for  M&S  is 
recognized  only  later.  Then,  the  results  of  the  M&S  come  too  late  to  benefit  the  overall 
system  design.  There  are  many  other  VV&A  issues  commonly  associated  with  non- 
compliance: 
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—  M&S  activities  aren’t  coordinated  with  acquisition  strategy. 

—  T&E  strategy  doesn’t  provide  validation  data. 

—  Models  aren’t  accredited  prior  to  use. 

—  M&S  isn’t  planned  throughout  acquisition  lifecycle. 

—  M&S  use  and  VV&A  strategy  aren’t  in  TEMP. 

—  Models  are  seldom  V&V’ed  -  and  if  done,  seldom  prior  to  use. 

—  M&S  and  VV&A  end-states  are  often  not  agreed  upon  at  program 
start,  resulting  in  lack  of  commitment  to  resource  them. 

—  Proprietary  models  cannot  be  VV&A’ed  without  contract  support. 


Stakeholder  [Kl: 

•  I  see  no  real  benefit  associated  with  non-compliance. 

•  Not  complying  with  COTF  instruction  would  cause  much  more  pain  than 
benefit  (if  any).  However,  PMs  should  find  ways  to  tailor  the  instruction  to  fit  within  the 
constraints  of  their  acquisition  programs  without  having  a  negative  effect  on  the  integrity 
of  the  VV&A  process. 

Stakeholder  [LI; 

•  I  see  no  real  benefit  associated  with  non-compliance. 

•  There  is  no  real  benefit  associated  with  non-compliance  whatsoever. 

Stakeholder  [Ml: 

•  I  see  a  few  minor  benefits  associated  with  non-compliance. 

•  A  few  minor  benefits  may  be  associated  with  non-compliance.  The  PM 
may  use  a  different  (nevertheless  valid)  methodology.  And  sometimes  it  may  be  more 
workable  to  complete  M&S  accreditation  after  the  model’s  use. 

Stakeholder  FN1: 

•  I  see  no  real  benefit  associated  with  non-compliance. 

Stakeholder  FOl : 

•  I  see  no  real  benefit  associated  with  non-compliance. 

•  Not  complying  with  the  instruction  means  that  proponents  will  not  be 
conducting  a  genuine  VV&A  process. 

Stakeholder  [PI: 

•  I  see  no  real  benefit  associated  with  non-compliance. 

QUESTION  7. 

What  do  you  see  to  be  the  costs  associated  with  non-compliance  with  this  instruction? 
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Table  A.7-1.  Choices  of  Score  Answers  Offered  to  Stakeholders  for  Question  7. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a  few 

no 

significant 

significant 

significant 

minor 

real 

costs 

costs 

cost 

costs 

cost 

Table  A.7-2.  Scores  Provided  by  Stakeholders  to  Question  #  7 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 
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P 

Score 

5 

5 

5 

4 

4 

3 

4 

4 

4 

5 

5 

3 

1 

4 

Comments  Provided  by  Stakeholders  for  Question  7: 

Stakeholder  [A1  |TA,Im}: 

•  I  see  many  significant  costs  associated  with  non-compliance. 

•  The  proponent  cannot  perform  adequately  the  needed  design  studies,  trade 
studies,  pre-test  planning,  pre-test  predictions,  and  post-test  analysis  when  the  M&S  is  of 
poor  quality,  as  would  be  indicated  by  its  failure  to  be  accredited.  One  cannot  trust  the 
results!  Consequently,  it  is  harder  to  analyze  and  solve  problems  early.  This  failure 
would  be  avoided  if  sufficient  funding  were  applied  to  M&S  development  and  its 
supporting  VV&A.  While  the  costs  associated  with  non-compliance  are  sometimes  hard 
to  quantify,  the  effects  are  nonetheless  apparent. 

Stakeholder  FBI : 

•  I  see  many  significant  costs  associated  with  non-compliance. 

•  M&S  that  does  not  meet  the  needs  of  the  Tester  /  Accreditor  does  not 
produce  useful  M&S,  i.e.,  it  does  not  provide  any  benefit.  Non-useful  M&S  would  bring 
the  program  to  a  halt. 

Stakeholder  [Cl: 

•  I  see  many  significant  costs  associated  with  non-compliance. 

•  The  many  significant  costs  of  non-compliance  relate  to  the  ultimate  costs 
of ‘doing  it  wrong’. 

Stakeholder  FD1: 

•  I  see  several  significant  costs  associated  with  non-compliance. 

•  The  costs  associated  with  non-compliance  depend  on  the  importance  of 
the  model  presented  to  the  Tester  /  Accreditor.  It  seems  to  be  anomalous  for  the 
proponent  to  develop  a  model  to  satisfy  an  OT  requirement  but  not  to  comply  with  the 
VV&A  process  needed  to  convince  the  Tester  /  Accreditor. 
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Stakeholder  [El: 

•  I  see  several  significant  costs  associated  with  non-compliance. 

•  The  costs  would  be  very  significant  as  they  would  be  equal  to  the  entire 
cost  to  develop  and  document  the  M&S,  since  the  M&S  would  be  rejected  in  its  entirety 
for  not  proving  to  be  fit  for  use  under  OT&E. 

Stakeholder  FG1: 

•  I  see  one  significant  cost  associated  with  non-compliance. 

•  In  the  case  of  one  ship  acquisition  program,  it  cost  $250M  (which  had  to 
be  taken  from  other  program  funds)  to  perform  the  physical  operational  test  that  the  M&S 
sought  to  accomplish  in  a  virtual  environment.  Perhaps  earlier  PM  interaction  with  the 
Tester  /  Accreditor  and  the  collection  of  some  additional  physical  (real)  data  would  have 
led  to  a  credible  M&S  solution. 

Stakeholder  [HI: 

•  I  see  several  significant  costs  associated  with  non-compliance. 

•  COTF  holds  the  Program  Managers  to  this  instruction.  Non-compliance 
could  cause  significant  change  in  the  acquisition  program’s  schedule  (and  thereby 
increase  program  costs).  Thus  the  program’s  assets  would  be  tied  up  and  would  need  to 
be  rescheduled. 

Stakeholder  [II: 

•  I  see  several  significant  costs  associated  with  non-compliance. 

•  Several  significant  costs  result  from  the  various  missed  opportunities  that 
non-compliance  puts  off,  including: 

—  Missed  opportunities  to  consider  early,  cost-effective  tradeoffs 
between  M&S,  T&E,  and  construction  techniques. 

—  Missed  opportunities  to  develop  complementary,  cost-effective 
strategies  to  conduct  T&E  and  M&S  to  achieve  the  objectives  of  T&E. 
M&S  (and  its  VV&A)  cannot  be  merely  thought  of  as  cheap  way  out 
of  testing. 

Stakeholder  [Kl: 

•  I  see  several  significant  costs  associated  with  non-compliance. 

•  The  Tester  /  Accreditor  could  fail  parts  of  the  PM’s  system  if  he  does  not 
comply  with  their  requirements  for  VV&A  because  they  would  not  have  the  necessary 
confidence  in  the  PM’s  M&S. 

Stakeholder  [LI: 

•  I  see  many  significant  costs  associated  with  non-compliance. 
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•  There  are  lots  of  process  costs  associated  with  non-compliance,  including 
the  loss  of  engineering  rigor,  accountability,  and  repeatability.  With  non-compliance, 
one  sees  different  programs  using  different  processes. 

•  The  financial  costs  of  non-compliance  are  evident  when  additional 
funding  is  required  to  build  up  the  PM’s  knowledge  base  to  the  level  that  has  been 
already  attained  within  COTF.  Through  compliance,  the  PM  can  leverage  the  knowledge 
of  the  Tester  /  Accreditor  with  little  additional  cost  to  achieve  a  successful  M&S. 

Stakeholder  [Ml: 

•  I  see  many  significant  costs  associated  with  non-compliance. 

•  The  PM  relies  on  the  Tester  /  Accreditor  to  accredit  the  model.  The 
funding  invested  in  the  M&S  is  wasted  without  accreditation  (accomplished  through  the 
VV&A  process). 

Stakeholder  FN1: 

•  I  see  one  significant  cost  associated  with  non-compliance. 

•  Not  following  the  VV&A  instruction  creates  a  significant  probability  of 
incurring  added  costs,  especially  when  trying  to  validate  and  verily  the  models  of 
federations.  On  what  basis  will  PMs  be  confident  that  the  ship  systems  they  are 
delivering  will  perfonn  the  way  they  are  predicting?  Physical  testing  will  have  to  be 
added  to  prove  the  VV&A  case.  This  is  necessary  because  no  one  can  buy  or  accept  a 
product  of  which  they  are  unsure,  especially  when  the  stakes  are  high.  The  costs  are 
ultimately  borne  by  the  warfighter. 

Stakeholder  FOl : 

•  I  see  no  real  cost  associated  with  non-compliance. 

•  There  is  no  real  cost  associated  with  non-compliance  if  VV&A  is  not 
done.  However,  the  cost  to  go  back  later  and  conduct  VV&A  will  be  much  more 
expensive. 

Stakeholder  [PI: 

•  I  see  several  significant  costs  associated  with  non-compliance. 

•  Non-compliance  with  the  instruction  would  continue  to  cost  the  U.S.  Navy 
more  money. 

QUESTION  8. 

What  do  you  see  to  be  the  risks  associated  with  non-compliance  with  this  instruction? 
Comments  Provided  by  Stakeholders  for  Question  8: 
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Table  A.8-1.  Choices  of  Score  Answers  Offered  to  Stakeholders  for  Question  8. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a  few 

no 

significant 

significant 

significant 

minor 

real 

risks 

risks 

risk 

risks 

risk 

Table  A.8-2.  Scores  Provided  by  Stakeholders  to  Question  #  8 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

5 

4 

5 

4 

4 

3 

4 

4 

4 

5 

5 

5 

5 

4 

Stakeholder  FA1  {TA,lm}: 

•  I  see  many  significant  risks  associated  with  non-compliance. 

•  The  proponent  cannot  perform  adequately  the  needed  design  studies,  trade 
studies,  pre-test  planning,  pre-test  predictions,  and  post-test  analysis  when  the  M&S  is  of 
poor  quality,  as  would  be  indicated  by  its  failure  to  be  accredited.  One  cannot  trust  the 
results!  Consequently,  it  is  harder  to  analyze  and  solve  problems  early.  This  failure 
would  be  avoided  if  sufficient  funding  were  applied  to  M&S  development  and  its 
supporting  VV&A.  While  the  risks  associated  with  non-compliance  are  sometimes  hard 
to  quantify,  the  effects  are  nonetheless  apparent. 

Stakeholder  FBI : 

•  I  see  several  significant  risks  associated  with  non-compliance. 

•  The  risks  are  that  the  requirements  of  the  M&S  will  not  be  met  and 
therefore  the  OT&E  test  requirements  would  not  be  fully  satisfied. 

Stakeholder  [Cl: 

•  I  see  many  significant  risks  associated  with  non-compliance. 

•  Models  should  be  a  level  of  fidelity  higher  (i.e.,  more  detailed  than)  what 
you  need.  This  would  lower  the  risk  that  a  detail  important  to  the  ship  system’s  predicted 
operational  performance  would  be  somehow  missed. 

Stakeholder  FD1: 

•  I  see  several  significant  risks  associated  with  non-compliance. 

•  There  are  very  significant  risks  associated  with  non-compliance.  The 
USN  still  has  to  accomplish  an  adequate  OT&E. 

Stakeholder  [El: 

•  I  see  several  significant  risks  associated  with  non-compliance. 
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•  There  are  significant  risks  with  non-compliance  because  OT&E 
requirements  have  to  be  satisfied. 

Stakeholder  TGI: 

•  I  see  one  significant  risk  associated  with  non-compliance. 

•  One  significant  risk  associated  with  non-compliance  was  recently  made 
evident  where  a  ship  acquisition  program  was  held  up  at  MS  B  while  resolving  the  issue 
of  whether  the  use  of  M&S  would  satisfy  one  of  the  OT  requirements. 

Stakeholder  [HI: 

•  I  see  several  significant  risks  associated  with  non-compliance. 

•  Several  significant  risks  are  associated  with  non-compliant  VV&A, 
including  the  need  to  “re-synchronize”  or  “re-coordinate”  the  acquisition  program’s 
“tracks”  (i.e.,  the  kind  of  activities  /  events  identified  in  the  TEMP).  As  this  would 
translate  to  cost  increases,  the  PM  would  have  to  go  back  to  the  Resource  Sponsor  for 
help.  But  the  Resource  Sponsor’s  solution  is  usually  to  advise  the  PM  to  “take  it  out  of 
hide”  and  reprogram  existing  funds  to  support  the  changes. 

Stakeholder  [II: 

•  I  see  several  significant  risks  associated  with  non-compliance. 

•  Several  significant  risks  result  from  the  various  missed  opportunities  that 
non-compliances  put  off,  including: 

—  Missed  opportunities  to  plan  the  M&S  (and  its  VV&A)  to  be  a 
continual  process  throughout  the  ship’s  acquisition  lifecycle,  which 
would  achieve  complete  success  for  use  of  the  M&S. 

—  Missed  opportunities  to  make  strategic,  up-front  investments  in  the 
M&S  (and  its  VV&A)  that  would  achieve  complete  success  for  the 
acquisition  program. 

—  Missed  opportunities  to  make  appropriate  replications  of  the  real  world 
in  the  M&S. 

•  The  devil  is  in  the  details,  which  tend  to  be  overlooked  when  PMs  don’t 
fully  comply  with  the  prescribed  process. 

Stakeholder  [Kl: 

•  I  see  several  significant  risks  associated  with  non-compliance. 

•  If  the  PM  does  not  comply  with  the  VV&A  requirements,  the  Tester  / 
Accreditor  could  disapprove  critical  performance  aspects  of  the  PM’s  system  since  they 
would  not  have  confidence  in  the  predictive  quality  of  the  M&S.  Poor  quality  M&S 
entails  erroneous  predictions  of  system  performance,  which  could  lead  to  poor  decisions 
that  adversely  impact  the  system’s  operators. 
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Stakeholder  [LI: 

•  I  see  many  significant  risks  associated  with  non-compliance. 

•  Getting  “false-positives”  are  the  risks  associated  with  non-compliance.  It 
can  be  presumed  that  the  PMs  are  moving  forward  in  good  faith  to  conduct  VV&A  on 
their  M&S.  However,  when  they  do  not  use  the  COTF  prescribed  process,  they  end  up 
having  to  develop  their  own.  This  will  result  in  unevenness  of  results  across  the  PMs. 
Some  PMs  will  be  more  stringent  than  COTF,  but  most  will  likely  be  less  stringent. 
Those  who  are  less  stringent  run  the  risk  of  getting  false-positive  results,  that  is,  of 
qualifying  an  M&S  that  is  really  deficient.  The  effects  of  that  false  positive  will  be 
rolled-up  into  the  system  to  which  the  M&S  is  applied.  The  resulting  disconnect  between 
expected  performance  and  actual  performance  means  “bad  news”  for  the  fleet. 

Stakeholder  [Ml; 

•  I  see  many  significant  risks  associated  with  non-compliance. 

•  There  is  little  difference  between  the  many  risks  and  the  many  costs 
associate  with  non-compliance.  They  really  go  hand-in-hand,  as  the  risks  imply  costs  and 
the  costs  reflect  risks.  Any  distinctions  between  the  two  relate  more  to  the  observer’s 
vantage  point. 

Stakeholder  FN1: 

•  I  see  many  significant  risks  associated  with  non-compliance. 

•  Not  following  the  VV&A  instruction  creates  a  significant  probability  of 
incurring  added  risks,  especially  when  trying  to  validate  and  verify  the  models  of 
federations.  On  what  basis  will  PMs  be  confident  that  the  ship  systems  they  are 
delivering  will  perform  the  way  they  are  predicting?  Physical  testing  will  have  to  be 
added  to  prove  the  VV&A  case.  This  is  necessary  because  no  one  can  buy  or  accept  a 
product  of  which  they  are  unsure,  especially  when  the  stakes  are  high.  The  risks  are 
ultimately  borne  by  the  warfighter. 

Stakeholder  FOl : 

•  I  see  many  significant  risks  associated  with  non-compliance. 

•  Non-compliance  with  the  instruction  entails  that  there  will  be  no 
confidence  in  the  M&S  proposed  for  use. 

Stakeholder  [PI: 

•  I  see  several  significant  risks  associated  with  non-compliance. 

•  There  are  several  significant  risks  associated  with  non-compliance;  the 
foremost  among  them  is  the  lack  of  credibility  for  those  models  developed  without  regard 
to  compliance. 
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QUESTION  9. 

To  what  extent  did  the  change  from  the  superceded  COMOPTEVFORINST  5000.1  (of  5 
September  1995)  to  the  new  COMOPTEVFORINST  5000.1  A  (of  9  September  2004) 
improve  or  degrade  the  VV&A  of  M&S  used  for  OT&E? 


Table  A.9-1.  Choices  of  Score  Answers  Offered  to  Stakeholders  for  Question  9. 


(5) 

(4) 

(3) 

(2) 

to 

significant 

minor 

no  real 

minor 

significant 

improvements 

improvements 

improvement 

degradations 

degradations 

Table  A.9-2.  Scores  Provided  by  Stakeholders  to  Question  #  9 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

3 

3 

5 

5 

5 

3 

5 

3 

3 

4 

4 

3 

4 

4 

Comments  Provided  by  Stakeholders  for  Question  9: 

Stakeholder  fAl  {TA,Im}: 

•  I  believe  no  real  improvement  to  VV&A  occurred. 

•  The  signed  version  of  the  5000.1  A  instruction  (which  excludes  much  of 
the  detailed  VV&A  guidance  that  is  provided  by  the  surface  warfare  VV&A  guide)  does 
an  adequate  job  of  specifying  the  “who”  “what”  “when”  “where”  and  “why”  of  VV&A 
but  tends  to  be  general  in  nature.  It  is  particularly  lacking  in  the  “how  to”  guidance  that 
the  PMs  seem  to  need.  A  more  thorough  VV&A  instruction  that  provides  greater  detail  is 
needed  -  one  which  could  be  tailored  to  match  the  specific  needs  of  each  program  - 
especially  in  today’s  environment  of  frequent  personnel  turnover,  both  in  government  and 
industry.  An  enhanced  instruction  would  promote  a  faster  learning  curve  for  the  PM. 
When  exceptions  to  the  normal  VV&A  process  need  to  be  made  (e.g.,  combining  the 
SMB  and  the  SCP),  stakeholders  can  nevertheless  follow  the  intent  of  the  VV&A 
guidance,  since  it  is  well  understood  and  the  programs  can  still  perform  all  the  tasks  that 
lead  to  high  quality  and  less  rework.  The  latest  draft  of  NMSO’s  VV&A  handbook 
provides  guidance  on  this  and  more.  The  OTDs  do  not  have  the  time  to  study  and 
become  experts  in  the  technical  details  of  M&S,  so  NMSO’s  VV&A  handbook  is  useful 
for  those  who  need  to  understand  more  about  M&S.  On  the  other  hand,  the  surface 
warfare  VV&A  guide  is  detailed,  but  only  in  the  bottom-line  management  needed  to 
conduct  the  VV&A. 

Stakeholder  fBl : 

•  I  believe  no  real  improvement  to  VV&A  occurred. 

•  The  old  instruction  had  much  more  detail,  with  appendices  and  guides. 
The  new  instruction  is  not  complete.  It  tends  to  be  too  broad.  Though  it  is  well  written, 
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it  does  not  tell  the  PM  what  products  COTF  expects  to  get.  Thus,  the  new  instruction 
does  not  provide  practical  guidance  so  much  as  a  very  good  over-arching  philosophy.  To 
make  up  for  the  detail  that  the  new  instruction  lacks,  the  OTDs  developed  a  Surface  Ship 
VV&A  Guide  that  provides  some  of  the  specific  “how  to”  information  that  had  been 
explicit  in  the  old  instruction.  (Thus,  some  of  the  historical,  working  experience  of  U.S. 
Navy  surface  ship  OT&E  M&S  VV&A  is  preserved.)  The  Surface  Ship  VV&A  Guide  is 
an  interim  solution.  A  revision  to  the  new  instruction  is  forthcoming  (which  may  be 
providing  the  needed  details  and  specific  guidance). 

Stakeholder  [Cl: 

•  I  believe  significant  improvements  to  VV&A  occurred. 

•  Significant  improvements  to  VV&A  guidance  are  seen  in  the  new 
instruction.  This  includes  the  insight  that  the  participants  (within  the  established 
accreditation  process)  should  move  from  a  focus  on  the  fidelity  of  the  model  to  the 
appropriateness  of  the  model  for  its  intended  use.  The  level  of  resolution  to  be  achieved 
for  the  model  depends  on  the  questions  being  asked. 

Stakeholder  FD1: 

•  I  believe  significant  improvements  to  VV&A  occurred. 

•  The  COMOPTEVFORINST  5000.1  (of  5  September  1995)  did  not  place 
great  emphasis  in  using  M&S  for  OT&E.  But  people’s  desire  to  participate  (i.e.,  to  use 
M&S  for  OT&E)  changed  from  1995  to  2004,  whereby  COMOPTEVFOR  INST 
5000.1  A  (of  9  September  2004)  was  issued.  The  keen  interest  in  using  (where 
practicable)  M&S  for  OT&E  was  expressed  by  the  CNO,  and  supported  by  COTF. 

•  The  change  from  the  superseded  instruction  to  the  new  instruction 
unquestionably  improved  the  VV&A  of  M&S  used  for  OT&E.  It  provides  a  far  better 
road  map  than  before. 

Stakeholder  [El: 

•  I  believe  significant  improvements  to  VV&A  occurred. 

•  Significant  improvements  to  VV&A  guidance  are  seen  in  the  new 
instruction.  With  the  old  instruction,  the  PMs  did  not  know  what  the  requirements  were. 

•  The  earlier  the  OTD  is  involved,  the  better  the  compliance.  The  new 
instruction  better  promotes  this. 

Stakeholder  FG1: 

•  I  believe  no  real  improvement  to  VV&A  occurred. 

•  No  real  improvement  to  VV&A  occurred  in  moving  to  the  new 
instruction.  It  is  not  clear  that  the  change  to  the  new  instruction  would  have  improved  the 
dynamic  observed  for  one  particular  surface  ship  VV&A  effort,  where  M&S  was 
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developed  to  address  a  missile  air  defense  requirement,  such  that  live-fire  testing  could  be 
avoided  for  events  within  a  certain  range  of  the  self-defense  test  ship.  Whether  invoking 
the  old  or  new  instruction,  the  key  issue  would  be  the  PM’s  early  interaction  with  COTF 
so  that  the  credibility  of  the  model  could  be  established  in  time  to  avoid  reprogramming 
funds  and  possible  delays  in  the  delivery  of  the  system. 

Stakeholder  [HI: 

•  I  believe  significant  improvements  to  VV&A  occurred. 

•  The  changes  incorporated  in  the  new  instruction  are  “golden,”  as  they 
allow  the  PM  more  flexibility  while  still  expecting  him  to  interact  with  COTF  early  in  the 
process  for  delivering  VV&A  products. 

Stakeholder  [II: 

•  I  believe  no  real  improvement  to  VV&A  occurred. 

•  The  new  instruction  kind  of  “hamstrings”  the  PM.  On  the  one  hand,  it 
more  clearly  articulates  COTF  VV&A  policy;  but  on  the  other  hand,  it  does  not  provide 
the  specific  expertise  (guidance)  the  PM  needs  to  be  able  to  implement  the  VV&A. 
While  PMs  can  obtain  the  necessary  practical  guidance  from  other  authorities  (e.g., 
NMSO),  it  is  important  that  COTF  (as  the  customer)  be  satisfied  with  the  process  and 
products  it  receives.  [Thus,  the  customer’s  practical  expectations  should  be  better 
expressed  in  a  revision  to  the  new  instruction.] 

•  Regardless  of  making  VV&A  instruction  changes,  real  improvement  to 
VV&A  would  be  achieved  if  PMs  were  to  place  more  emphasis  on  addressing  the 
technical  challenges  of  M&S,  which  include: 

—  Reducing  model  setup  time.  (This  can  be  accomplished  by  CAD 
conversion  for  ship  models.) 

—  Running  high  fidelity  models  in  real-time. 

—  End-to-end  Family  of  Systems  (FOS)  testing.  (An  example  would  be 
the  “Testing  in  a  Joint  Environment  Initiative.”) 

—  Enterprise  M&S  efforts. 

—  “No  degrade”  regression  testing. 

—  Emphasizing  a  “physics  based”  approach  to  VV&A.  (Specific  areas 
are:  surrogate  vice  threat  testing;  measuring  and  modeling  independent 
variables;  contractor  (tactical)  software  testing;  and  end-to-end 
testing.) 

—  Recognizing  where  physics  does  not  provide  the  basis  for  the  VV&A. 
(Examples  are:  “Netcentric  Warfare  Knowledge  Sharing”  and  “Social 
Network  Performance  and  Measurement.”) 


Stakeholder  [Kl: 

•  I  believe  no  real  improvement  to  VV&A  occurred. 
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•  No  real  improvement  to  VV&A  seems  to  have  occurred  in  the  move  to  the 
new  instruction.  It  is  difficult  to  detect  any  change  in  VV&A  process  outcomes  without 
performing  a  detailed  assessment,  which  requires  time. 

Stakeholder  [LI ; 

•  I  believe  minor  improvements  to  VV&A  occurred. 

•  Minor  improvements  to  VV&A  occurred  in  the  move  to  the  new 
instruction.  For  one,  the  new  instruction  provides  a  sense  of  structure,  fonnat,  and 
process.  Combined  with  the  ready  advice  and  assistance  of  the  OTDs  at  COTF,  the  new 
instruction  guides  the  PM  through  the  steps  of  the  VV&A  process  with  only  minor 
difficulties. 

Stakeholder  [Ml: 

•  I  believe  minor  improvements  to  VV&A  occurred. 

•  The  new  instruction  better  articulates  accepted  VV&A  standards  and 
provides  more  specific  guidance  than  the  general  philosophy  of  the  old  instruction. 

Stakeholder  FN1: 

•  I  believe  no  real  improvement  to  VV&A  occurred. 

•  No  real  improvement  to  VV&A  occurred  in  the  move  to  the  new 
instruction  (but  it  would  be  difficult  to  see  any  change  as  the  new  instruction  has  not  been 
in  effect  that  long.). 

Stakeholder  jOj: 

•  I  believe  minor  improvements  to  VV&A  occurred. 

•  The  new  instruction  is  more  user-friendly  and  shorter. 

Stakeholder  [PI: 

•  I  believe  minor  improvements  to  VV&A  occurred. 

•  Some  minor  improvements  seem  to  have  resulted  from  the  move  to  the 
new  instruction  (but  it  is  difficult  to  specifically  identify  them). 

QUESTION  10. 

What  improvements  would  you  recommend  for  COMOPTEVFOR’s  Accreditation 
process  (beyond  that  prescribed  by  the  new  COMOPTEVFORINST  5000.1  A)? 

Comments  Provided  by  Stakeholders  for  Question  10: 
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Table  A.10-1.  Choices  of  Score  Answers  Offered  to  Stakeholders  for  Question  10. 


(5) 

(4) 

(3) 

(2) 

(1) 

many 

several 

one 

a  few 

no 

significant 

significant 

significant 

minor 

real 

improvements 

improvements 

improvement 

improvements 

improvement 

Table  A.10-2.  Scores  Provided  by  Stakeholders  to  Question  #  10 

Stakeholder 

A 

B 

C 

D 

E 

G 

H 

I 

K 

L 

M 

N 

O 

P 

Score 

4 

5 

1 

2 

4 

3 

1 

5 

3 

4 

2 

1 

4 

1 

Stakeholder  FA1  {TA,Im}: 

•  I  recommend  several  significant  improvements  for  COMOPTEVFOR's 
Accreditation  process. 

•  There  are  several  significant  improvements  that  can  be  made  to 
COMOPTEVFOR's  Accreditation  process.  Among  these  are: 

—  Incorporate  the  “how  to”  guidance  of  the  Surface  Warfare  VV&A 
Guide  in  to  the  COTF  VV&A  instruction. 

—  Make  the  language  more  understandable  to  the  average  (and 
sometimes  novice)  user. 

—  Make  sure  the  process  diagram  and  the  mock  schedule  from  the 
Surface  Warfare  VV&A  Guide  are  included  in  the  COTF  VV&A 
instruction  because  these  two  diagrams  provide  a  grand  summary  of 
the  entire  process.  Instruction  users  could  readily  refer  to  these  two 
pages  and  quickly  formulate  plans  regarding  requirements  tasking, 
organization,  planning  and  reporting  documents,  assessment  and 
accreditation  requirements,  as  well  as  the  timing  (schedule)  required  to 
perform  all  these  tasks. 

—  Incorporate  the  format  and  content  examples  that  are  currently 
provided  at  the  end  of  the  Surface  Warfare  VV&A  Guide. 

•  Since  OT  is  the  final  step  before  full  production,  it  would  be  most  efficient 
to  offer  the  PM  a  VV&A  instruction  that  contains  both  policy  and  detailed  “how  to” 
guidance  a  single,  usable  resource.  The  PM  would  there  find  efficient  methods  to 
organize,  estimate  costs,  determine  schedules,  and  finish  a  quality  M&S  on  time  and 
within  budget. 

Stakeholder  FBI : 

•  I  recommend  many  significant  improvements  for  COMOPTEVFOR's 
Accreditation  process. 
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•  There  are  several  significant  improvements  that  can  be  made  to 
COMOPTEVFOR's  Accreditation  process.  Among  these  are: 

—  An  appendix  should  be  added  to  the  instruction  that  would  provide 
detailed  examples  of  how  to  conduct  VV&A.  The  appendix  should 
provide  the  new  PM  a  step-by-step  model  of  how  to  go  through  the 
VV&A  process.  Such  a  guide  would  enable  PMs  to  better  comply. 

—  Incorporate  the  VV&A  tools  provided  by  the  Surface  Warfare  VV&A 
Guide,  as  this  would  increase  the  effectiveness  of  PMs. 

—  Review  the  forthcoming  new  SECNAV  VV&A  instruction  and  derive 
/  incorporate  the  practical  guidance  that  would  apply  to  the  VV&A  of 
M&S  used  for  OT&E  into  the  COTF  VV&A  instruction. 

—  Place  greater  emphasis  on  Tester  /  Accreditor  participation  on  the 
SCP,  as  this  would  foster  the  early  interactions  needed  to  achieve 
compliance. 

—  Review  when  and  where  (e.g.,  milestone  decisions  and  review  boards) 
VV&A  is  required  and  approved.  Avoid  imposing  overly  burdensome 
requirements. 

—  Adequately  plan  for  and  fund  the  personnel  needed  to  review  the 
documentation  produced  by  the  SCP  and  other  points  in  the  VV&A 
process. 

—  Provide  a  real  “hammer”  for  the  VV&A  to  be  done. 

—  Increase  “ground-level”  involvement  in  M&S  and  its  VV&A. 

—  Make  the  commitment  to  establish  a  quality  approach  to  the  VV&A 
effort.  Program  the  needed  funding  and  hire  /  employ  qualified,  smart, 
and  disciplined  personnel  to  accomplish  the  job. 


Stakeholder  [Cl: 

•  I  recommend  no  real  improvement  for  COMOPTEVFOR's  Accreditation 
process. 

•  There  are  no  real  improvements  that  can  be  made  to  COTF’s 
Accreditation  process.  The  key  issue  for  improving  the  accreditation  process  regards  the 
implementation  of  VV&A  and  not  necessarily  what  the  VV&A  requires  by  way  of 
procedure.  Instead  of  focusing  on  the  process  for  producing  documentation,  the 
participants  should  focus  on  the  technical  content  conveyed  in  the  documentation.  In 
other  words,  the  instruction  should  not  emphasize  how  to  do  conduct  the  VV&A  process 
or  require  close  adherence  to  a  checklist.  Rather,  the  focus  should  be  on  the 
appropriateness  of  the  model  to  answer  the  technical  question  related  to  its  intended  use. 
In  sum,  the  most  prominent  “content”  issues  for  consideration  are: 

—  VV&A  participants  should  move  from  a  focus  on  the  fidelity  of  the 
model  to  the  appropriateness  of  the  model  for  its  intended  use. 

—  Models  should  be  developed  at  one  level  of  fidelity  more  detailed  than 
what  the  proponent  would  otherwise  need. 

—  VV&A  participants  should  improve  their  formal  understanding  of  the 
model’s  intended  use. 


148 


Stakeholder  [D1 : 

•  I  recommend  a  few  minor  improvements  for  COMOPTEVFOR's 

Accreditation  process. 

•  There  are  a  few  minor  improvements  that  can  be  made  for 

COMOPTEVFOR's  Accreditation  process.  Upon  initial  review,  there  is  nothing  to 
reconsider  regarding  the  new  instruction.  In  moving  from  the  old,  it  succeeded  in 
capturing  a  new,  needed  perspective.  Now,  it  should  be  noted  that  the  new  instruction  is 
a  philosophically  related  instruction.  The  forthcoming  SECNAV  VV&A  Handbook  is 
expected  to  provide  more  guidance  on  how  to  conduct  VV&A. 

Stakeholder  [El: 

•  I  recommend  a  few  minor  improvements  for  COMOPTEVFOR's 

Accreditation  process. 

•  Among  the  possible  improvements  to  be  made  is  finding  a  way  to  make 
models  more  available  for  reuse  -  both  within  the  USN  and  among  all  the  Military 
Services.  Testing  in  the  Joint  Force  Environment  relies  on  a  having  an  adequately 
modeled  virtual  environment.  More  funding  is  needed  to  fully  support  this  initiative, 
along  with  a  technical  commitment  at  and  above  any  particular  Service’s  ability.  Then 
the  question  can  be  asked,  “What  does  a  PM  expect  to  be  able  to  do  when  he  wants  to 
reuse  the  M&S  developed  by  another,  only  to  find  that  this  PM  is  not  following  the 
Service’s  VV&A  process?” 

Stakeholder  FG1: 

•  I  recommend  one  significant  improvement  for  COMOPTEVFOR's 

Accreditation  process. 

•  Perhaps  another  revision  of  the  instruction  could  be  crafted  to  help  the  PM 
understand  more  clearly  that  unplanned  funding  requirements  could  be  avoided  if  earlier 
interactions  with  the  Tester  /  Accreditor  are  sought  under  the  VV&A  process. 

Stakeholder  [HI: 

•  I  recommend  no  real  improvement  for  COMOPTEVFOR's  Accreditation 
process. 

•  There  is  not  enough  information  to  see  whether  any  real  improvement 
could  be  made  by  another  revision. 

Stakeholder  [II: 

•  I  recommend  many  significant  improvements  for  COMOPTEVFOR's 
Accreditation  process. 
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•  There  are  a  number  of  important  “lessons  learned”  from  recent  VV&A 
experience  that  can  be  emphasized  or  incorporated  as  guidance  (etc.)  in  an  improved 
COTF  accreditation  process,  as  follows: 

—  M&S  is  not  just  a  “tool”  but  a  process.  The  products  are  an  outcome 
of  the  process.  (That  M&S  is  commonly  viewed  as  a  tool  (only)  for 
acquisition  programs  to  satisfy  T&E  requirements  explains  why  M&S 
reuse  is  less  than  10%.)  The  goal  of  M&S  is  to  facilitate  good 
acquisition.  The  objective  of  VV&A  is  to  produce  useful  M&S. 

—  M&S  and  VV&A  activities  have  to  be  integral  part  of  program  and  test 
process  from  inception.  The  intended  purpose  of  the  M&S  must  be 
clear. 

—  M&S  must  have  at  least  a  CMMI  level  3  process  maturity  to  avoid 
VV&A  problems. 

—  Tradeoffs  between  M&S,  T&E,  and  construction  techniques  should  be 
considered  early  on. 

—  M&S  and  VV&A  must  be  a  continual  process  throughout  an 
acquisition  lifecycle  to  be  successful. 

—  M&S  and  VV&A  cannot  be  thought  of  as  cheap  way  out  of  testing. 

—  M&S  requires  up  front  investment  to  achieve  success. 

•  The  bottom  line  is  that  the  M&S  proponent  should  do  what  is  technically 
right.  Properly  perfonned  VV&A  will  show  this  to  all  stakeholders. 

Stakeholder  [Kl: 

•  I  recommend  one  significant  improvement  for  COMOPTEVFOR's 
Accreditation  process. 

•  The  practical  application  of  any  process  is  a  difficult  thing  to  address,  no 
less  improve.  That  being  said,  it  seems  that  the  new  instruction  can  provide  more 
guidance  to  the  PM  on  the  practical  application  of  the  VV&A  process. 

Stakeholder  [LI: 

•  I  recommend  several  significant  improvements  for  COMOPTEVFOR's 
Accreditation  process. 

•  Of  the  several  significant  improvements  that  can  be  made  to 
COMOPTEVFOR's  Accreditation  process,  one  that  would  be  innovative  to  the  PM  is  the 
deployment  of  an  on-line  process.  Were  such  a  capability  implemented  via  a  distributed 
knowledge  base,  it  would  allow  for  VV&A  participants  to  work  the  process  without  the 
cost  of  traveling  to  remote  sites  for  meetings. 

•  Another  improvement  would  be  to  implement  “check  sheets.”  The  current 
VV&A  reporting  requirements  are  way  too  complex.  It  seems  that  some  activities  see 
their  participation  in  VV&A  as  a  way  to  expend  labor-hours  and  deliberately  make  the 
process  complex  as  a  means  to  keep  them  in  business.  Sometimes  it  appears  that  solving 
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the  technical  challenges  of  M&S  is  easier  than  managing  the  VV&A  business  process, 
with  its  myriad  of  meetings  and  reports.  Some  kind  of  reform  is  needed  to  simplify  these 
complexities. 

•  The  role  and  function  of  NMSO  with  respect  to  supporting  the  PM’s 
efforts  to  comply  with  the  instruction  could  be  clarified  in  a  revision  to  the  COTF 
instruction. 

Stakeholder  [Ml: 

•  I  recommend  a  few  minor  improvements  for  COMOPTEVFOR's 
Accreditation  process. 

•  Discussion  related  to  model  reuse  should  be  incorporated  in  a  future 
update  of  COMOPTEVFOR's  VV&A  instruction.  This  would  be  helpful  in  conveying 
the  Tester  /  Accreditor’s  standards  and  expectations  for  reuse. 

Stakeholder  FN1: 

•  I  recommend  no  real  improvement  for  COMOPTEVFOR's  Accreditation 
process. 

•  No  call  for  real  improvement  to  COMOPTEVFOR's  Accreditation  process 
comes  to  mind. 

Stakeholder  FOl : 

•  I  recommend  several  significant  improvements  for  COMOPTEVFOR's 
Accreditation  process. 

•  There  are  several  improvements  that  can  be  made  to  the  COTF  VV&A 
instruction,  including  the  following  items: 

—  Reference  the  new  Navy  Standard,  SND32. 

—  Add  process  diagrams. 

—  Provide  more  detail  regarding  M&S  requirements  and  VV&A 
acceptability  criteria. 

—  Describe  the  expected  relationship  between  the  PMs  and  COTF  (the 
Tester  /  Accreditor). 


Stakeholder  [PI: 

•  I  recommend  no  real  improvement  for  COMOPTEVFOR's  Accreditation 
process. 

•  COMOPTEVFOR's  Accreditation  process  does  not  have  need  of 
improvement.  The  new  instruction  should  be  supported  as  is.  Rather  than  look  for  any 
improvement  in  the  COTF  instruction,  the  U.S.  Navy  should  focus  its  efforts  on 
standardizing  the  process  described  by  SECNAVINST  5200.40A.  [Supporting  this 
instruction  would  actually  resolve  the  majority  of  compliance  issues  experienced  with  the 
COTF  VV&A  instruction.]  Attention  should  also  be  made  to  the  update  and 
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implementation  of  NMSO’s  VV&A  Handbook.  The  resulting  standardized  process  for 
VV&A  is  where  proponents  will  see  real  cost  benefits. 


OVERVIEW  OF  STAKEHOLDERS’S  SCORES. 

Table  A.  10-3,  below,  provides  an  overview  of  the  scores  provided  by  the 
stakeholders  interviewed  for  questions  #1  through  #10. 


Table  A.10-3.  Overview  of  Stakeholders’  Scores 

Question: 

#1 

#2 

#3 

#4 

#5 

#6 

#7 

#8 

#9 

#10 

ID 

Stakeholder 

Group 

Sr. 

Score 

Score 

Score 

Score 

Score 

Score 

Score 

Score 

Score 

Score 

A 

Tester/ 

Accreditor 

Lower 

5 

5 

5 

4 

3 

3 

5 

5 

3 

4 

B 

Tester/ 

Accreditor 

Lower 

5 

5 

5 

4 

1 

1 

5 

4 

3 

5 

C 

Tester/ 

Accreditor 

Middle 

5 

4 

5 

1 

2 

1 

5 

5 

5 

1 

D 

Tester/ 

Accreditor 

Upper 

5 

1 

4 

2 

1 

1 

4 

4 

5 

2 

E 

Tester/ 

Accreditor 

Upper 

5 

1 

5 

2 

2 

1 

4 

4 

5 

4 

G 

Resource 

Sponsor 

Middle 

1 

1 

4 

2 

2 

1 

3 

3 

3 

3 

H 

Resource 

Sponsor 

Lower 

2 

5 

4 

2 

2 

1 

4 

4 

5 

1 

I 

Resource 

Sponsor 

Upper 

2 

5 

5 

1 

1 

1 

4 

4 

3 

5 

K 

Program 

Manager 

Lower 

4 

4 

3 

2 

3 

1 

4 

4 

3 

3 

L 

Program 

Manager 

Middle 

4 

5 

5 

3 

1 

1 

5 

5 

4 

4 

M 

Program 

Manager 

Lower 

4 

4 

4 

1 

1 

2 

5 

5 

4 

2 

N 

Program 

Manager 

Middle 

2 

4 

5 

4 

2 

1 

3 

5 

3 

1 

O 

Program 

Manager 

Upper 

5 

5 

5 

4 

2 

1 

1 

5 

4 

4 

P 

VV&A 

Standards 

Upper 

4 

5 

4 

1 

1 

1 

4 

4 

4 

1 
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